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Committee  on  Armed  Services,  U.S. 

Senate 


Accountability  Integrity  Reliability 

Highlights 


Why  GAO  Did  This  Study 

For  several  decades,  Congress  and 
the  Department  of  Defense  (DOD) 
have  explored  ways  to  improve  the 
acquisition  of  major  weapon 
systems,  yet  program  outcomes 
and  their  underlying  causes  have 
proven  resistant  to  change.  Last 
year,  we  reported  that  the 
cumulative  cost  growth  in  DOD’s 
portfolio  of  major  programs  was 
$296  billion.  The  opportunity  to 
achieve  meaningful  improvements 
may  now  be  at  hand  with  the 
recent  introduction  of  major 
reforms  to  the  acquisition  process. 

In  response  to  a  mandate  from  this 
Committee,  GAO  has  issued  several 
reports  about  DOD’s  budget  and 
requirements  processes  to  support 
weapon  program  stability.  This 
follow-on  report  focuses  on  (1) 
identifying  weapon  programs  that 
are  achieving  good  outcomes,  (2) 
the  factors  that  enable  some 
programs  to  succeed,  and  (3) 
lessons  to  be  learned  from  these 
programs  to  guide  implementation 
of  recent  reforms.  GAO  analyzed 
DOD’s  portfolio  of  major  defense 
programs  and  conducted  case 
study  reviews  of  five  programs. 


What  GAO  Recommends 


While  no  new  recommendations 
are  being  made,  previous  GAO 
recommendations  have  been 
incorporated  into  recent  reforms. 

In  this  report,  we  present  lessons 
learned  to  help  effectively 
implement  these  reforms.  In 
written  comments,  DOD  noted  that 
it  has  recently  instituted  several 
major  changes  to  acquisition  policy 
that  are  aimed  at  starting  programs 
right. 

View  GAO-10-522  or  key  components. 

For  more  information,  contact  Michael  J. 
Sullivan  at  (202)  512-4841  or 
sullivanm@gao.gov. 


What  GAO  Found 

While  GAO’s  work  has  revealed  significant  aggregate  cost  and  schedule 
growth  in  DOD’s  portfolio  of  major  defense  acquisition  programs,  individual 
programs  within  the  portfolio  vary  greatly  in  terms  of  cost  growth  and 
schedule  delays.  Our  analysis  of  individual  program  performance  found  that 
21  percent  of  programs  in  DOD’s  2008  major  defense  acquisition  portfolio 
appeared  to  be  stable  and  on  track  with  original  cost  and  schedule  goals. 

These  programs  tended  to  represent  relatively  smaller  investments,  with  just 
under  9  percent  of  total  dollars  invested  in  these  programs.  Programs  that 
appeared  to  be  on  track  were  markedly  newer  and  had  development  cycles 
that  were  shorter  than  highly  unstable  programs. 

The  stable  programs  we  studied  were  supported  by  senior  leadership,  run  by 
disciplined  program  managers,  and  had  solid  business  cases  that  were  well- 
executed.  These  programs  benefited  from  strong  leadership  support,  in  some 
cases  because  the  programs  were  perceived  as  having  an  immediate  need  and, 
therefore,  were  viewed  as  a  higher  priority  by  senior  leaders.  Their  program 
managers  tended  to  share  key  attributes  such  as  experience,  leadership 
continuity,  and  communication  skills  that  facilitated  open  and  honest  decision 
making.  As  a  result,  these  programs  established  sound,  knowledge-based 
business  plans  before  starting  development  and  then  executed  those  plans 
using  disciplined  approaches.  They  pursued  evolutionary  or  incremental 
acquisition  strategies,  leveraged  mature  technologies,  and  established  realistic 
cost  and  schedule  estimates  that  accounted  for  risk.  They  were  able  to  invest 
in  early  planning  and  systems  engineering,  and  made  trade-offs  to  close  gaps 
between  customer  needs  and  available  resources  to  arrive  at  a  set  of 
requirements  that  could  be  developed  within  cost  and  schedule  targets.  After 
approval,  the  programs  resisted  new  requirements  and  maintained  stable 
funding.  These  practices  are  in  contrast  to  prevailing  pressures  to  force 
programs  to  compete  for  funds  by  exaggerating  achievable  capabilities, 
underestimating  costs,  and  assuming  optimistic  delivery  dates. 

Congress  and  DOD  have  taken  major  steps  toward  reforming  the  defense 
acquisition  system  that  may  increase  the  likelihood  weapon  programs  succeed 
in  meeting  their  planned  cost  and  schedule  objectives.  Many  of  these  steps  are 
consistent  with  key  elements  in  our  case  study  analysis.  In  particular,  the  new 
DOD  policy  and  legislative  provisions  place  greater  emphasis  on  front-end 
planning  and  establishing  sound  business  cases  for  starting  programs.  For 
example,  the  provisions  strengthen  systems  engineering  and  cost  estimating, 
and  require  early  milestone  reviews,  prototyping,  and  preliminary  designs. 
They  are  intended  to  enable  programs  to  refine  a  weapon  system  concept  and 
make  cost,  schedule,  and  performance  trade-offs  before  significant 
commitments  are  made.  Fundamentally,  the  provisions  should  help  programs 
replace  risk  with  knowledge,  and  set  up  more  executable  programs.  If  reform 
is  to  succeed,  however,  programs  that  present  realistic  strategies  and  resource 
estimates  must  succeed  in  winning  approval  and  funding. 
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United  States  Government  Accountability  Office 
Washington,  DC  20548 


May  6,  2010 

The  Honorable  Carl  Levin 
Chairman 

The  Honorable  John  McCain 
Ranking  Member 
Committee  on  Armed  Services 
United  States  Senate 

For  several  decades,  Congress  and  the  Department  of  Defense  (DOD)  have 
been  exploring  ways  to  improve  the  acquisition  of  major  weapon  systems, 
yet  poor  program  outcomes  and  their  underlying  causes  have  proven 
resistant  to  change.  Last  year,  we  reported  that  the  cumulative  cost  growth 
in  DOD’s  portfolio  of  96  major  defense  acquisition  programs  was  $296 
billion  from  first  estimates,  and  the  average  delay  in  delivering  promised 
capabilities  to  the  warfighter  was  22  months. 1  However,  in  recent  years 
Congress  and  DOD  have  made  changes  to  policies  which  have  led  to 
improvements  in  some  aspects  of  the  acquisition  process.  For  example, 
our  2010  assessment  of  major  weapon  system  acquisition  programs  found 
that  programs  started  since  2006  have  higher  levels  of  technology  maturity 
when  they  begin  system  development.2  More  importantly,  the  opportunity 
to  achieve  widespread  and  meaningful  improvements  in  weapon  system 
programs  may  now  be  at  hand,  as  both  Congress  and  DOD  have  recently 
introduced  major  reforms  to  the  defense  acquisition  process,  including 
changes  to:  improve  the  department’s  ability  to  balance  requirements  with 
resources;  establish  a  stronger  foundation  for  starting  programs;  and 
execute  programs  more  effectively.  While  these  changes  are  promising, 
the  challenge  to  achieving  better  program  outcomes  will  be  not  only  to 
ensure  that  they  are  consistently  put  into  practice,  but  also  to  confront  the 
environment  in  DOD  that  has  made  the  weapon  acquisition  area  resistant 
to  reform. 

In  the  Senate  Armed  Services  Committee  report  for  the  2006  National 
Defense  Authorization  Act,  the  committee  directed  us  to  review  DOD’s 
budget  and  requirements  processes  and  assess  how  these  processes  can 
better  support  program  stability  in  major  weapon  system  acquisition.  In 


^AO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-09-326SP 
(Washington,  D.C.:  March  30,  2009). 

2GAO,  Defense  Acquisitions:  Assessments  of  Selected  Weapon  Programs,  GAO-10-388SP 
(Washington,  D.C.:  March  30,  2010). 
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response  to  this  mandate,  we  have  issued  several  reports  that  highlight 
weaknesses  in  these  processes.3  This  report,  the  last  in  response  to  this 
mandate,  identifies  practices  already  present  in  the  acquisition 
environment  that  are  contributing  to  stable  acquisition  programs  and 
yielding  good  outcomes.  Lessons  learned  from  successful  weapon 
programs  may  provide  additional  guidance  for  programs  implementing 
recent  acquisition  reforms.  Specifically,  we  (1)  identified  and  described 
programs  within  DOD’s  major  weapon  system  acquisition  program 
portfolio  that  were  stable  and  on  track  to  meet  cost  and  schedule  targets 
outlined  at  program  development  start;  (2)  determined  what  factors 
enabled  some  stable  programs  to  achieve  these  cost  and  schedule  targets; 
and  (3)  analyzed  recent  acquisition  reform  initiatives  to  determine  how 
lessons  learned  from  these  stable  programs  can  be  of  use  as  DOD 
implements  acquisition  reform. 

To  conduct  our  work,  we  identified  stable  programs  in  DOD’s  portfolio  of 
major  weapon  system  acquisition  programs  by  analyzing  data  from  the 
December  2007  Selected  Acquisition  Reports  (SAR),  the  most  recent  full 
reports  issued  by  DOD  as  of  the  time  we  conducted  our  work,  as  well  as 
other  program  data.4  We  determined  each  program’s  growth  in 
development  cost,  unit  cost,  and  schedule  from  its  original  program 
baseline  estimate  and  categorized  the  program  as  “stable,”  “moderately 


3In  March  2007,  we  reported  that  DOD  lacks  an  effective,  integrated  approach  to  balancing 
its  weapon  system  investments  with  available  resources.  In  July  2008,  we  reported  that  a 
knowledge-based  funding  approach  could  improve  major  weapon  system  program 
outcomes.  In  September  2008,  we  reported  that  DOD’s  requirements  determination  process 
had  not  been  effective  in  prioritizing  joint  capabilities.  See  GAO,  Best  Practices:  An 
Integrated  Portfolio  Management  Approach  to  Weapon  System  Investments  Could 
Improve  DOD’s  Acquisition  Outcomes,  GAO-07-388  (Washington,  D.C.:  March  30,  2007); 
Defense  Acquisitions:  A  Knoivledge-Based  Funding  Approach  Could  Improve  Major 
Weapon  System  Program.  Outcomes,  GAO-08-619  (Washington,  D.C.:  July  2,  2008);  and 
Defense  Acquisitions:  DOD’s  Requirements  Determination  Process  Has  Not  Been 
Effective  in  Prioritizing  Joint  Capabilities,  GAO-08-1060  (Washington,  D.C.:  September 
25,  2008). 

4DOD  did  not  issue  complete  Selected  Acquisition  Reports  for  its  major  defense  acquisition 
programs  in  2009,  so  program  costs  had  not  been  updated  since  the  December  2007 
reports,  as  of  the  time  we  conducted  our  work. 
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unstable,”  or  “highly  unstable.”5  From  among  the  more  stable  programs, 
and  other  programs  identified  as  successful  by  acquisition  experts,  we 
selected  the  following  five  programs  as  case  studies:  the  Army’s  High 
Mobility  Artillery  Rocket  System  (HIMARS);  the  Air  Force’s  Joint  Direct 
Attack  Munition  (JDAM)  and  Small  Diameter  Bomb  (SDB);  and  the  Navy’s 
Poseidon  Multi-Maritime  Aircraft  (P-8A)  and  STANDARD  Missile-6  (SM-6). 
For  each  case  study,  we  reviewed  key  documents,  and  interviewed  past 
and  present  program  officials  to  identify  key  factors  contributing  to  the 
program’s  stability.  We  also  leveraged  prior  GAO  work  where  we  had 
identified  enablers  of  stability  in  other  programs,  such  as  the  Navy’s 
F/A-18E/F  Super  Hornet  and  the  Air  Force’s  F-16  Fighting  Falcon,  as  well 
as  causes  of  instability  in  unstable  programs,  such  as  the  Air  Force’s  F-22 
Raptor  and  Global  Hawk  programs.  We  reviewed  recent  legislative  and 
policy  changes  relating  to  defense  acquisitions.  More  information  about 
our  scope  and  methodology  is  provided  in  appendix  I.  We  conducted  this 
performance  audit  from  November  2008  to  May  2010,  in  accordance  with 
generally  accepted  government  auditing  standards.  Those  standards 
require  that  we  plan  and  perform  the  audit  to  obtain  sufficient,  appropriate 
evidence  to  provide  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives.  We  believe  that  the  evidence  obtained 
provides  a  reasonable  basis  for  our  findings  and  conclusions  based  on  our 
audit  objectives. 


Background 


Over  the  last  several  years,  our  work  has  highlighted  a  number  of 
underlying  systemic  causes  for  cost  growth  and  schedule  delays  in 
weapon  programs.  At  the  strategic  level,  DOD’s  processes  for  identifying 
warfighter  needs,  allocating  resources,  and  managing  acquisitions,  which 
together  define  its  weapon  system  investment  strategy,  do  not  work 
together  effectively.  As  a  result,  the  department  often  fails  to  balance  the 
competing  needs  of  the  warfighter  and  commits  to  more  programs  than 
available  resources  can  support.  At  the  program  level,  DOD’s  culture  and 


5We  classified  programs  as  “stable,”  “moderately  unstable,”  or  “highly  unstable”  based  on 
growth  in  research,  development,  test  and  evaluation  (RDT&E)  and  program  acquisition 
unit  cost  (PAUC)  estimates,  as  well  as  schedule  slip,  from  their  Milestone  B  estimates. 
Programs  were  assessed  based  on  whether  their  RDT&E  growth  was  less  than  10  percent, 
between  10  and  35  percent,  or  35  percent  or  more;  whether  their  PAUC  growth  was  less 
than  10  percent,  between  10  and  30  percent,  or  30  percent  or  more;  and  whether  their  initial 
capability  schedule  slip  was  less  than  6  months,  between  6  and  12  months,  or  12  months  or 
longer.  We  classified  a  program  as  stable  if  it  was  below  the  lowest  threshold  in  at  least 
two  of  our  cost  and  schedule  measures  and  no  higher  than  the  middle  threshold  range  in 
the  third  measure. 
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environment  often  allow  programs  to  start  with  too  many  unknowns.  In 
other  words,  programs  enter  the  acquisition  process  without  a  full 
understanding  of  requirements;  with  cost  and  schedule  estimates  based  on 
overly  optimistic  assumptions;  and  with  insufficient  knowledge  about 
technology,  design,  and  manufacturing. 

Prior  GAO  work  in  best  practices  has  emphasized  the  importance  of 
having  a  sound  business  case  when  starting  major  defense  programs.  The 
business  case  in  its  simplest  form  is  demonstrated  evidence  that  (1)  the 
identified  needs  are  real  and  necessary  and  can  best  be  met  with  the 
chosen  weapon  system  concept  and  (2)  the  chosen  concept  can  be 
developed  and  produced  within  existing  resources — funding,  time, 
technologies,  and  people.  In  the  DOD  weapon  system  acquisition 
environment,  a  business  case  is  typically  established  at  Milestone  B  when 
significant  resources  are  committed.  Programs  are  then  measured  against 
the  business  case  established  at  Milestone  B. 

A  primary  reason  for  cost  and  schedule  problems  is  the  encouragement 
within  the  acquisition  environment  of  overly  ambitious  and  lengthy 
product  developments — sometimes  referred  to  as  “revolutionary”  or  “big 
bang”  acquisition  programs — that  embody  too  many  technical  unknowns 
and  insufficient  knowledge  about  performance  and  production  risks.  The 
knowledge  gaps  are  largely  the  result  of  a  lack  of  early  and  disciplined 
systems  engineering  analysis  of  a  weapon  system’s  requirements  prior  to 
beginning  system  development  which  translates  customer  needs  into  a 
producible  weapon  system.  If  this  early  systems  engineering  is  not 
performed,  as  has  often  been  the  case  with  DOD’s  major  acquisitions  in 
the  past,  significant  cost  increases  can  occur  as  the  system’s  requirements 
become  better  understood  by  the  government  and  contractor. 

With  high  levels  of  uncertainty  about  requirements,  technologies,  and 
design,  program  cost  estimates  and  their  related  funding  needs  are  often 
understated,  effectively  setting  programs  up  for  cost  and  schedule  growth. 
We  recently  assessed  both  service  and  independent  cost  estimates  for  20 
major  weapon  system  programs  and  found  that  while  the  independent 
estimates  were  somewhat  higher,  both  estimates  were  too  low  in  most 
cases.6  In  some  programs,  cost  estimates  were  off  by  billions  of  dollars. 
Estimates  this  inaccurate  do  not  provide  the  necessary  foundation  for 
sufficient  funding  commitments.  The  programs  we  reviewed  frequently 


0GAO-O8-619. 
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lacked  sufficient  knowledge  and  detail  to  develop  sound  cost  estimates. 
Without  this  knowledge,  cost  estimators  may  fail  to  adequately  account  for 
risk  and  uncertainty  because  they  are  relying  on  overly  optimistic 
assumptions. 

Recognizing  the  need  for  more  discipline  in  weapon  system  acquisition, 
Congress  and  the  department  have  been  making  changes  to  policies  in 
recent  years  and  we  have  seen  improvements  in  performance  across  some 
aspects  of  the  acquisition  process.  Most  recently,  the  acquisition  reforms 
that  took  place  in  2009  indicate  a  desire  on  the  part  of  Congress  for 
improvement  and  willingness  on  the  part  of  the  department  to  change  its 
culture  and  achieve  better  outcomes,  and  address  some  of  the  sources  of 
instability  that  we  have  identified  in  our  prior  work. 


Some  2008  Major 
Defense  Acquisition 
Programs  Appeared 
to  Be  on  Track  to 
Meet  Their  Original 
Cost  and  Schedule 
Projections 


While  our  work  has  revealed  significant  aggregate  cost  and  schedule 
growth  in  DOD’s  major  defense  acquisition  program  (MDAP)  portfolio, 
individual  programs  within  the  portfolio  vary  greatly  in  the  extent  to  which 
they  experience  cost  and  schedule  growth.  In  our  analysis  of  63  individual 
programs  and  subprograms  in  DOD’s  2008  MDAP  portfolio,  we  found  that 
21  percent  appeared  to  be  stable  and  on  track  to  meet  original  cost  and 
schedule  projections.  These  stable  programs  entailed  relatively  small 
investments  and  had  shorter  development  cycles  than  programs  reporting 
substantial  cost  growth  and  schedule  slip.  They  were  also  significantly 
newer  than  less  stable  programs. 

Approximately  21  percent,  or  13,  of  the  63  MDAP  programs  that  filed 
December  2007  SARs  and  were  at  least  3  years  into  development  at  the 
time7  were  stable,  meaning  they  appeared  to  be  on  track  to  end  close  to 


We  excluded  from  our  analysis  23  programs  for  which  development  cost,  unit  cost,  or 
initial  operational  capability  data  were  not  available.  We  also  excluded  9  MDAPs  that  were 
less  than  3  years  into  development  as  of  December  2007.  We  have  previously  found  that  a 
significant  portion  of  cost  increases  often  do  not  occur  until  after  a  program  is 
approximately  halfway  through  its  development  cycle.  The  average  projected  development 
cycle  time  for  the  programs  in  the  2008  MDAP  portfolio  was  approximately  7  years.  We 
therefore  determined  that  including  in  our  analysis  programs  that  were  less  than  3  years 
into  development  might  artificially  overstate  stability  in  the  portfolio.  Because  we  did  not 
conduct  any  analysis  of  the  excluded  programs,  we  do  not  have  any  basis  on  which  to 
comment  upon  the  stability  of  the  excluded  programs. 


Page  5 


GAO-10-522  Defense  Acquisitions 


the  cost  and  schedule  estimates  established  at  development  start.8  These 
13  programs  are  shown  in  table  1  below. 


Table  1 :  Stable  Programs 

Program 

Service 

Development 
cost  growth 
(percent) 

Unit  cost 
growth 
(percent) 

Delay  in  initial 
capability 
(months) 

Programs  in  Production  as  of  December  2007 

EA-18G  Growler  electronic  warfare  aircraft 

Navy 

3.8 

4.7 

0 

Minuteman  III  Propulsion  Replacement  Program 

Air  Force 

-6.1 

6.0 

0 

National  Airspace  System 

Air  Force 

5.4 

8.8 

5 

Small  Diameter  Bomb  Increment  1 

Air  Force 

-4.9 

-14.3 

-1 

Programs  in  Development  as  of  December  2007 

AGM-88E  Advanced  Anti-Radiation  Guide  Missile 

Navy 

6.2 

0.0 

4 

E-2D  Advanced  Hawkeye  surveillance  aircraft 

Navy 

5.8 

9.6 

0 

Family  of  Advanced  Beyond  Line-of-SightTerminals 

Air  Force 

0.9 

9.0 

0 

Mobile  User  Objective  System 

Navy 

6.5 

-1.2 

6 

Navy  Multiband  Terminals 

Navy 

-1.6 

-4.8 

0 

P-8A  Multi-mission  Maritime  Aircraft 

Navy 

-3.9 

0.6 

0 

PATRIOT/Medium  Extended  Air  Defense  System  Combined 
Aggregate  Program  Fire  Unit 

Army 

-5.6 

-4.4 

0 

PATRIOT/Medium  Extended  Air  Defense  System  Combined 
Aggregate  Program  Missile 

Army 

-10.2 

-3.3 

0 

STANDARD  Missile-6 

Navy 

-7.7 

-3.7 

0 

Source:  GAO  analysis  of  DOD  data. 

We  assessed  another  24  of  these  63  programs  as  moderately  unstable,  and 
the  remaining  26  as  highly  unstable. 

The  13  programs  that  we  assessed  as  stable  tended  to  be  smaller, 
representing  just  under  9  percent  (or  $103  billion  in  fiscal  year  2009 


8While  we  attempted  to  account  for  our  previous  finding  that  cost  increases  often  occur 
late  in  development  by  excluding  younger  programs  from  our  analysis,  it  is  likely  that  some 
programs  that  appeared  stable  according  to  this  data  may  have  since  exhibited  instability. 
For  instance,  the  E-2D  Advanced  Hawkeye  aircraft  program,  which  appeared  to  be  on  track 
to  meet  original  cost  and  schedule  estimates  as  of  December  2007  (54  months  into  its 
development),  reported  a  critical  cost  breach  in  2009. 
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dollars)  of  the  total  $1.15  trillion  estimated  total  cost  for  these  programs. 
(See  fig.  1.) 


Figure  1 :  Distribution  of  DOD  Acquisition  Programs  by  Number  of  Programs  and  Dollar  Value 
By  number  of  programs  By  dollar  value  of  programs 


Stable 


Moderately  unstable 


Highly  unstable 


Source:  GAO  analysis  of  DOD  data. 


Stable 


Moderately  unstable 


Highly  unstable 


The  average  total  program  cost  of  the  stable  programs  was  $7.9  billion, 
while  the  averages  were  $17.2  billion  and  $24.5  billion  for  moderately  and 
highly  unstable  programs,  respectively. 

Thirty-eight  of  the  programs  we  analyzed  had  completed  their 
development  cycles  and  started  production  by  December  2007.  The  four 
stable  programs  in  this  group  tended  to  have  shorter  development  cycles, 
on  average  more  than  2  years  shorter  than  the  average  for  highly  unstable 
programs.  (See  fig.  2.) 
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Figure  2:  Average  Length  of  Development  Cycle  for  36  Programs  in  Production  (in 
months) 

Average  months  in  development 
100 


Source:  GAO  analysis  ofDOD  data. 


Note:  This  analysis  excludes  two  satellite  programs,  because  those  acquisitions  have  been 
conducted  under  DOD's  space  acquisition  process  ratherthan  its  weapons  acquisition  process,  and 
therefore  the  M  ilestone  C  dates  listed  for  these  programs  are  not  necessarily  indicative  of  the  start  of 
production,  as  they  are  for  other  programs. 

The  programs  that  we  assessed  as  stable  were  also  markedly  newer  than 
the  less  stable  programs.  Ten — or  77  percent — of  the  13  stable  programs 
had  started  development  just  3  to  5  years  prior  to  the  December  2007 
reporting  date,  compared  to  only  25  percent  of  the  moderately  unstable 
programs  and  4  percent  of  the  highly  unstable  ones  that  started  in  the 
same  timeframe.  (See  fig.  3  for  breakdown  of  stable  programs’  years  since 
development  start.) 
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Figure  3:  Distribution  of  Stable  Programs  by  Years  Since  Development  Start 

Number  of  programs 
12 


3  to  5  5  to  10  10  or 

more 

Years  since  Milestone  B 

Source:  GAO  analysis  ofDOD  data. 


Stable  Programs  Had 
Strong  Leadership 
Support,  Disciplined 
Program  Managers, 
and  Executable 
Business  Cases 


The  stable  programs  we  studied  had  strong  senior  leadership  support, 
disciplined  program  managers,  and  solid  business  plans  which  were  well- 
executed.  (Figure  4  presents  a  notional  illustration  of  the  relationship  of 
these  key  factors.)  These  programs  benefited  from  strong  leadership 
support,  in  some  cases  because  the  programs  were  perceived  as  having  an 
immediate  need  and,  therefore,  were  viewed  as  a  higher  priority  by  senior 
leaders  in  the  services  and  DOD.  Their  program  managers  tended  to  share 
key  attributes  such  as  experience,  leadership  continuity,  and 
communication  skills  that  facilitated  open  and  honest  decision  making.  As 
a  result,  these  programs  established  sound,  knowledge-based  business 
plans  before  starting  development  and  then  executed  those  plans  using 
disciplined  approaches.  They  pursued  incremental  acquisition  strategies, 
leveraged  mature  technologies,  and  established  realistic  cost  and  schedule 
estimates  that  accounted  for  risk.  They  were  able  to  invest  in  early 
planning  and  systems  engineering,  and  made  trade-offs  to  close  gaps 
between  customer  needs  and  available  resources  to  arrive  at  a  set  of 
requirements  that  could  be  developed  within  cost  and  schedule  targets. 
After  approval,  the  programs  resisted  new  requirements  and  maintained 
stable  funding.  These  practices  are  in  contrast  to  prevailing  pressures  in 
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DOD  that  force  programs  to  compete  for  funds  by  exaggerating  achievable 
capabilities,  underestimating  costs,  and  assuming  optimistic  delivery 
dates. 


Figure  4:  Key  Factors  That  Enable  Program  Success 


While  we  found  13  programs  that  were  able  to  maintain  stability  and  stay 
on  track  toward  good  outcomes,  we  focused  in  depth  on  the  details  of  5 
programs  to  identify  reasons  for  their  success.  These  were  the  Air  Force’s 
Small  Diameter  Bomb  and  Joint  Direct  Attack  Munition  programs,  the 
Navy’s  STANDARD  Missile-6  and  Poseidon  Multi-Maritime  Aircraft  (P-8A) 
programs,  and  the  Army’s  High  Mobility  Artillery  Rocket  System  program. 9 
Table  2  summarizes  cost  and  schedule  outcomes  of  each  of  our  case  study 
programs. 


9Three  of  our  five  case  studies  are  among  the  13  stable  programs  we  identified;  however, 
the  other  two  had  moderate  cost  and/or  schedule  growth  but  were  selected  because  they 
had  other  attributes  that  were  valuable  to  study.  In  particular,  we  selected  JDAM  as  a  case 
study  because  its  unit  costs  were  lower  than  originally  estimated  and  it  is  widely 
recognized  within  the  defense  acquisition  community  as  a  successful  program.  HIMARS 
delivered  capability  on-time  and  it  was  one  of  the  Army’s  most  stable  programs  in  the 
portfolio. 


Page  10 


GAO-10-522  Defense  Acquisitions 


Table  2:  Case  Study  Programs 

Program 

Service 

Success  indicators  as  of  December  2007 

Small  Diameter  Bomb  (SDB) 
Increment  1 

Air  Force 

SDB  reduced  development  costs  by  almost  5  percent  and  unit  costs  by  more  than  14 
percent,  and  made  the  system  available  one  month  earlier  than  anticipated. 

J  oint  Direct  Attack  Munition 
(J  DAM) 

Air  Force 

J  DAM  reduced  unit  cost  by  25  percent  which,  in  part,  enabled  the  Air  Force  to  purchase 
more  than  twice  the  number  of  units  originally  expected. 

STANDARD  Missile-6  (SM-6) 
Extended  Range  Active  Missile 

Navy 

The  SM-6  program  was  on  track  to  reduce  expected  development  costs  by  more  than  7 
percent  and  unit  costs  by  almost  4  percent,  and  expected  to  deliver  initial  capability  on 
schedule. 

P-8A  Multi-mission  Maritime 
Aircraft 

Navy 

P-8A  was  on  track  to  reduce  estimated  development  costs  by  almost  4  percent  with  less 
than  1  percent  increase  in  unit  cost  and  was  scheduled  to  deliver  initial  capability  on- 
time. 

High  Mobility  Artillery  Rocket 
System  (HIMARS) 

Army 

HIMARS  delivered  initial  capability  on  time  in  2005.  While  development  cost  for  the 
program  grew  about  20  percent  from  original  estimates,  this  largely  reflects  a 
subsequent  need  to  up-armor  the  vehicle  in  order  to  face  new  threats  in  the  wars  in  Iraq 
and  Afghanistan. 

Source:  GAO  analysis  of  DOD  data. 

In  the  stable  programs  we  studied,  we  found  that  strong,  consistent 
support  from  DOD  and  service  leadership  fostered  the  planning  and 
execution  of  solid  business  plans  while  helping  program  managers  to 
adapt  to  the  inevitable  program  perturbations.  Program  managers  of  stable 
and  successful  programs  were  able  to  make  knowledge-based,  disciplined 
decisions  from  the  start  and  resist  pressure  to  overreach  or  add 
requirements  because  of  this  strong  institutional  support.  In  some  cases, 
high-level  support  may  have  been  especially  strong  due  to  the  particular 
circumstances  of  the  programs,  such  as  being  designated  as  a  high 
priority.  For  example: 

Early  in  the  planning  for  SDB,  the  Air  Force  Chief  of  Staff  established  a 
clear  written  “Commander’s  Intent”  stating  that  his  priority  was  to  have 
the  weapon  system  available  for  use  by  the  end  of  2006.  According  to 
program  officials,  having  this  priority  from  the  top  was  invaluable  in 
helping  the  program  establish  and  maintain  an  effective  business  plan  for 
acquiring  the  system.  For  example,  as  the  SDB  operational  requirements 
were  going  through  the  DOD  requirements  determination  process,  the 
Joint  Staff  attempted  to  add  more  requirements.  In  particular,  the  Army 
wanted  SDB  to  have  additional  protections  from  nuclear  attack;  however, 
the  program  office  got  support  from  the  user  community,  which, 
buttressed  by  support  from  the  Office  of  the  Secretary  of  Defense, 
managed  to  keep  nuclear  hardening  from  becoming  a  requirement  by 
arguing  effectively  that  it  did  not  make  sense  for  the  weapon.  In  addition, 


Strong  Leadership  Support 
and  Unique  Circumstances 
Were  Key  to  Achieving 
Stable  Programs 
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SDB  was  designated  as  the  Air  Force’s  highest  priority  program  at  Eglin 
Air  Force  Base,  and  program  officials  told  us  SDB  was  given  preference 
for  resources  above  all  other  programs  on-base.  Furthermore,  SDB  was 
designated  as  a  “pathfinder  program” — a  group  of  programs  established  to 
pioneer  ways  to  gain  speed  and  credibility  in  acquisition. 

•  According  to  program  officials,  the  termination  of  the  STANDARD  Missile- 
2  (SM-2)  Block  IVA  program  due  to  cost,  schedule,  and  performance 
problems  prompted  the  Navy  to  modify  the  next  in  its  series  of  planned 
standard  missile  programs.  Initially,  the  Navy  proposed  the  STANDARD 
Missile-5  (SM-5)  program,  which  was  intended  to  develop  sophisticated 
targeting  capabilities.  However,  strong  support  from  senior  acquisition 
leaders  allowed  program  officials  to  advocate  for  a  more  achievable  and 
affordable  “80  percent  solution”  which  resulted  in  the  SM-6  program. 
According  to  an  early  SM-6  program  manager,  the  urgent  need  for  a 
successful  program  helped  set  the  stage  for  conducting  thorough  and 
detailed  planning  work  prior  to  Milestone  B.  In  addition,  the  program 
received  strong  support  from  the  Assistant  Secretary  of  the  Navy  for 
Research,  Development  and  Acquisition  in  obtaining  full  funding  for  a 
realistic,  risk-based  cost  estimate. 

•  According  to  an  early  program  manager,  although  a  mission  need  for 
HIMARS  was  identified  in  the  early  1990s,  the  program  was  not  started 
right  away  due  to  a  lack  of  funding.  However,  prototypes  of  the  system 
were  developed  through  an  Advanced  Concept  Technology  Demonstration 
(ACTD)  project — a  DOD  process  to  get  new  technologies  that  meet  critical 
military  needs  into  the  hands  of  users  faster  and  at  less  cost — and 
successfully  demonstrated  by  the  Army.  This  early  program  manager  told 
us  HIMARS  had  been  dubbed  “a  70  kilometer  sniper  rifle”  by  troops  who 
have  used  the  system  in  theater,  because  of  its  range  and  accuracy.  Based 
on  this  success  and  the  cancellation  at  the  time  of  the  Army’s  Crusader 
artillery  program,  HIMARS  took  on  greater  importance  for  the  Army. 

•  JDAM  was  conceived  of  as  a  program  that  would  integrate  Global 
Positioning  System  (GPS)  technologies  into  a  precision  strike  guidance 
system.  JDAM  was  designated  as  one  of  five  high-profile  programs  to  be 
executed  under  the  Defense  Acquisition  Pilot  Project,  which  gave  JDAM 
the  authority  to  implement  certain  acquisition  reform  provisions  under  the 
Federal  Acquisition  Streamlining  Act  of  1994  before  they  were  published 
in  regulation.  According  to  studies  of  the  JDAM  program,  advocates  for 
the  acquisition  reform  movement  wanted  quick,  highly  visible  wins  from 
these  pilot  programs  to  demonstrate  the  benefits  of  the  legislation’s 
strategy.  JDAM  program  officials  have  stated  that  this  sense  of  urgency 
and  the  increased  flexibility  provided  to  these  pilot  programs  allowed 
officials  to  move  quickly  to  achieve  doable  requirements.  For  instance, 
JDAM  officials  were  able  to  streamline  the  program’s  milestone  review 
process  and  reporting  procedures.  In  addition,  according  to  program 
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officials,  the  program  allowed  the  contractor  to  maintain  configuration 
control  and  use  commercial  products  and  make  use  of  various  commercial 
acquisition  procedures  that  were  uncommon  at  the  time,  with  the  goal  of 
lowering  costs.  Also,  the  program  was  freed  to  structure  innovative 
solutions  such  as  a  20-year  warranty  included  in  the  unit  price. 
Furthermore,  according  to  an  early  program  manager  for  JDAM,  the 
program  was  given  very  specific  direction  by  the  Air  Force  Chief  of  Staff 
to  develop  and  produce  the  bomb  at  a  low  unit  cost.  This  support  allowed 
the  program  to  make  cost,  schedule,  and  performance  trade-offs  and  arrive 
at  a  solid  business  case  for  delivering  the  weapon  system.  According  to  the 
early  program  manager,  one  year  into  the  program,  a  senior  Air  Force 
general  pushed  JDAM  to  accelerate  its  delivery  schedule.  However,  when 
the  program  manager  asked  the  general  if  this  had  been  cleared  with  the 
Air  Force  Chief  of  Staff,  the  subject  was  dropped. 

•  The  primary  goal  for  the  P-8A  program  was  to  replace  the  capabilities  of 
the  P-3C  Orion,  an  aging  Navy  patrol  aircraft  that  is  very  important  to 
maritime  security  objectives,  but  is  beginning  to  experience  significant 
structural  fatigue  problems  and  must  be  replaced.  According  to  program 
officials,  this  sense  of  immediacy  has  heightened  the  P-8A’s  priority  and 
has  forced  the  Navy  leadership  to  set  realistic  requirements  for  quick 
delivery.  However,  an  early  program  manager  said  that,  rather  than 
rushing  into  system  development,  the  leadership  permitted  the  program  to 
conduct  a  detailed  planning  phase  and  establish  a  well-balanced  package 
of  cost,  schedule  and  performance. 

Beyond  our  case  studies,  we  have  also  seen  examples  in  the  past  where 
strong  support  from  high  level  officials  and  unique  circumstances 
contributed  to  program  success.  DOD  acquisition  officials,  for  example, 
noted  that  senior  Navy  leadership  may  have  made  the  decision  to  pursue  a 
cautious,  more  evolutionary  approach  for  development  of  the  F/A-18E/F 
Super  Hornet  tactical  aircraft,  in  part  because  of  (1)  embarrassment  over 
the  much  publicized  failure  of  the  A-12  stealth  attack  aircraft  program 
which  was  terminated  after  considerable  investment  when  the  Navy 
determined  that  the  contractor  was  unable  to  develop  and  deliver  an 
aircraft  that  met  requirements;  and  (2)  diminishing  capacity  of  the 
aircraft’s  previous  generation — the  F/A-18  C/D  Hornet.  In  contrast,  an 
official  observed  that  the  F-22  Raptor  fighter  aircraft  program  took  a 
significantly  more  revolutionary,  risky  approach,  likely  due,  in  part,  to  the 
fact  that  the  legacy  F-15  Eagle  fighter  aircraft  were  still  performing  so 
well. 
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Stable  Programs  Had 
Strong  Program  Managers 
Who  Shared  Key  Attributes 


In  addition  to  support  from  the  top,  program  managers  from  successful 
programs  tended  to  have  similar  attributes  for  success  such  as  experience, 
leadership  continuity,  and  communication  skills  that  facilitated  open  and 
honest  decision  making.  These  program  managers  were  empowered  to 
make  good  decisions,  allowing  them  to  be  accountable  for  the  success  or 
failure  of  the  program.  We  found  that  successful  managers  took  proactive 
measures  to  ensure  the  stability  of  their  programs,  including  reaching  out 
to  stakeholders  in  the  user  and  testing  communities  to  facilitate  their 
collaboration.  For  example,  one  program  manager  was  described  in  a 
lessons  learned  memo  developed  by  program  officials  as  “part  technical 
expert,  part  bulldog,  and  part  diplomat.  Steeped  in  the  technical  details  of 
weapon  development  and  aircraft  integration,  he  sniffed  out  and  pre¬ 
empted  technical  risks,  made  quick  decisions,  and  aptly  convinced 
stakeholders  to  support  his  positions.” 

Officials  from  our  case  study  programs  indicated  that  prior  experience 
gives  a  program  manager  the  knowledge  to  recognize  and  mitigate  risks, 
and  effectively  respond  to  unanticipated  problems  that  arise.  For  example, 
an  early  program  manager  for  SM-6  told  us  that  he  and  other  key  staff  had 
been  involved  in  the  previous  STANDARD  Missile  program  (SM-2  Block 
IVA)  that  had  been  terminated.  He  explained  that  they  were  therefore  very 
clear  on  what  the  potential  problems  were  and  were  highly  motivated  to 
try  to  avoid  them  if  possible.  In  addition,  the  current  program  manager  of 
SM-6  said  he  has  been  in  the  STANDARD  Missile  program  office  since  the 
mid-1980s  and  that  he  had  worked  in  the  program  in  various  capacities 
before  becoming  program  manager. 

In  addition,  our  case  study  programs  experienced  continuity  of  program 
management  and  other  key  staff  that  assisted  knowledge-based  decision 
making.  We  have  previously  reported  that  frequent  program  manager 
turnover  may  promote  shortsightedness,  challenge  continuity,  and  reduce 
accountability  for  poor  outcomes.  In  2007,  we  reported  that,  for  programs 
started  since  March  2001,  the  average  program  manager  tenure  was  only 
1.4  years.10  In  contrast,  managers  for  our  five  case  study  programs  served 
an  average  of  2.4  years.  For  instance,  the  SDB  program  had  one  program 
manager  who  served  from  before  Milestone  B  through  the  low-rate 
production  decision,  providing  leadership  continuity  for  the  entire  system 
development  phase.  Also,  many  of  our  case  study  programs  told  us  that 


10GAO,  Defense  Acquisitions:  Department  of  Defense  Actions  on  Program.  Manager 
Empowerment  and  Accountability ,  GAO-08-62R  (Washington,  D.C.:  November  9,  2007). 
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other  key  staff,  such  as  senior  engineers,  served  many  years  in  the 
program  office  and  provided  continuity  and  information  necessary  for 
knowledge-based  decision  making.  JDAM  officials  also  noted  that  the 
continuity  of  key  civil  service  and  contractor  personnel  has  proven  very 
beneficial  because  several  other  personnel  have  left  the  program  due  to 
military  deployments  and  reassignments.  Specifically,  this  included  the 
chief  engineer,  who  has  been  with  the  program  from  the  beginning,  and 
the  director  of  production  and  delivery,  who  has  been  called  upon  to 
perform  as  the  JDAM  program  manager  to  cover  6-  to  9-month 
deployments  of  several  past  military  program  managers.  Several  support 
contractors  also  have  been  with  the  JDAM  program  for  many  years. 

While  the  admission  of  program  difficulties  may  be  seen  as  detrimental  to 
a  program  manager’s  career,  leaders  of  our  case  study  programs 
understood  that  direct  and  candid  communication  are  essential  to 
program  success.  Program  officials  observed  that,  by  fostering  a 
reputation  for  honesty  and  leadership,  they  were  able  to  develop 
credibility  with  stakeholders,  including  contractors,  and  make  compelling 
cases  for  what  was  needed.  They  emphasized  the  importance  of  including 
stakeholders  early  on  in  the  process  of  solving  problems  so  they  are 
invested  in  the  solutions  being  developed.  For  example,  one  early  program 
manager  for  P-8A  explained  that  his  cooperative  relationship  with  the 
requirements  community  enabled  them  to  speak  with  a  united  voice  about 
the  need  to  keep  requirements  achievable.  He  described  their  approach  as 
“starting  slow  to  go  fast”  as  opposed  to  the  more  common  approach  of 
“rushing  to  failure.”  In  addition,  he  noted  that  candid  communication  and  a 
welcoming  attitude  for  scrutiny  fostered  top  cover  support  from  senior 
leadership  in  the  Pentagon.  Together,  support  from  senior  leadership  and 
the  requirements  community  for  an  extended  concept  development  phase 
permitted  the  program  to  invest  in  success  upfront,  and  this  support 
carried  throughout  the  execution  of  the  program.  Also,  SDB  officials 
explained  that  an  early  program  manager  proactively  established 
collaborative  relationships  with  organizations  that  the  program  relied  on. 
For  example,  the  program  manager  sought  early  buy-in  from  the  testing 
community  on  the  program’s  test  and  evaluation  plans  which  helped 
facilitate  keeping  the  testing  schedule  on  track  later  on.  The  program 
office  also  cultivated  very  effective  communication  with  its  contractor, 
assigning  program  office  representatives  to  work  as  facilitators  with 
competing  contractors  during  pre-Milestone-B  prototyping  work. 

Similarly,  the  SM-6  program  manager  said  being  on  a  first-name  basis  with 
the  contractor  vice-presidents  helps  him  to  manage  the  program 
effectively.  SM-6  program  officials  also  stated  that  the  contractor  program 
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manager  meets  weekly  with  staff  to  ensure  the  best  possible  talent  is 
working  on  his  program. 


Stable  Programs 
Established  Sound, 
Knowledge-Based 
Business  Plans  at  Program 
Start  and  Executed  with 
Discipline 


The  stable  programs  we  studied  exhibited  the  key  elements  of  a  sound 
knowledge-based  business  plan  at  program  development  start.  These 
programs  pursued  capabilities  through  evolutionary  or  incremental 
acquisition  strategies,  had  clear  and  well-defined  requirements,  leveraged 
mature  technologies  and  production  techniques,  and  established  realistic 
cost  and  schedule  estimates  that  accounted  for  risk.  They  then  executed 
their  business  plans  in  a  disciplined  manner,  resisting  pressures  for  new 
requirements  and  maintaining  stable  funding. 


Stable  Programs  Had  The  programs  we  reviewed  typically  took  an  evolutionary  acquisition 

Achievable  Increments  Based  approach,  addressing  capability  needs  in  achievable  increments  that  were 
on  Well-Defined  Requirements  based  on  well-defined  requirements.  To  determine  what  was  achievable, 

the  programs  invested  in  systems  engineering  resources  early  on  and 
generally  worked  closely  with  industry  to  ensure  that  requirements  were 
clearly  defined.  Performing  this  up-front  requirements  analysis  provided 
the  knowledge  for  making  trade-offs  and  resolving  performance  and 
resource  gaps  by  either  reducing  the  proposed  requirements  or  deferring 
them  to  the  future.  The  programs  were  also  grounded  in  well-understood 
concepts  of  how  the  weapon  systems  would  be  used.  For  example: 


•  According  to  program  officials,  SDB  was  designed  to  meet  a  pressing  Air 
Force  need  for  a  low-collateral-damage  weapon  which  was  small  enough 
to  maximize  the  number  that  can  be  carried  aboard  a  single  aircraft. 
Although  the  Air  Force  initially  wanted  SDB  to  have  capability  for  hitting 
both  fixed  and  mobile  targets,  a  decision  was  made  early  on  to  defer  the 
more  difficult  mobile  target  capability  to  a  later  program  (SDB  II). 
According  to  the  Air  Force’s  Deputy  for  Acquisition,  the  program  worked 
early  on  with  the  warfighter  community  to  focus  on  the  “art  of  the 
possible,”  enabling  them  to  proceed  with  an  evolutionary  acquisition 
approach.  An  analysis  of  alternatives  was  conducted  which  considered  a 
broad  range  of  alternatives  and  adequately  assessed  their  risks.  According 
to  program  officials,  prior  to  Milestone  B,  the  contractor  submitted  system 
performance  specifications  which  were  incorporated  into  the  development 
contract.  Once  requirements  were  finalized  at  Milestone  B,  they  were 
limited,  and  the  program  had  a  firm  understanding  of  what  compromises 
had  been  made  and  what  capability  has  been  deferred.  Program  officials 
told  us  they  did  not  want  to  take  on  extra  cost  and  schedule  risk  to  try  to 
achieve  more  than  was  possible. 
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•  According  to  program  officials,  in  planning  the  P-8A  program,  the  Navy 
limited  requirements  to  capabilities  “as  good  as”  those  of  the  P-3C  and 
deferred  additional  capabilities  to  later  increments,  adopting  an  “open 
architecture”  approach  to  enable  this  acquisition  strategy.  The  program 
also  began  with  a  robust  analysis  of  alternatives  which  included  a 
commercial  derivative  approach  and  helped  the  Navy  recognize  that  an 
unmanned  aircraft  platform  could  perform  some  of  the  mission  thus 
decreasing  P-8A’s  requirements.  The  program  received  feedback  from 
competing  contractors  on  proposed  requirements  to  make  sure  the 
requirements  were  well-understood  before  Milestone  B.  This  feedback 
resulted  in  some  requirement  modifications  before  the  award  of  the 
development  contract. 

•  HIMARS  was  designed  to  be  a  wheeled  version  of  a  heavier  tracked  missile 
launcher — the  M270 — and  lightweight  enough  to  be  transported  on  a  C-130 
aircraft.  The  design  and  requirements  were  well-understood  and  realistic 
from  the  outset,  with  rapid  transportability  of  the  platform  a  key  goal  and 
weight  reduction  a  key  challenge,  according  to  program  officials.  Program 
officials  also  stated  that  there  was  a  drive  to  use  as  much  existing 
hardware  as  possible  on  the  program,  and  maximize  commonality  with  the 
most  recent  variant  of  the  M270  launcher. 

•  The  JDAM  program  was  initially  laid  out  in  three  phases  to  increase 
capabilities,  including  an  adverse  weather  precision  strike  capability  based 
on  a  need  that  was  demonstrated  during  Operation  Desert  Storm.  Phase  I 
had  clear,  well-defined  requirements — to  add  existing  GPS  technology  to 
an  existing  inventory  of  “dumb”  warheads.  The  final  phase  was  based  on 
technologies  to  be  developed  over  time  in  the  science  and  technology 
environment  before  they  were  integrated  into  the  acquisition  program. 
According  to  JDAM  program  officials,  communication  with  the  user 
community  while  they  were  in  the  planning  phase  allowed  trade-offs 
which  resulted  in  considerable  cost  avoidance.  The  program  had  an 
extensive  user  trade-off  program  during  the  initial  requirements 
development  in  which  the  users  and  contractors  adjusted  their 
requirements  and  designs  to  accommodate  each  other.  The  program 
treated  cost  as  a  key  performance  parameter — the  iterative  planning 
process  cut  unit  costs  by  more  than  half. 

•  SM-6  is  the  next  generation  in  the  STANDARD  Missile  program  and  has 
added  extended  range  and  active  missile  seeker  homing  capabilities  for 
improved  flight  responsiveness  and  guidance  over  previous  generations. 
According  to  program  officials,  it  is  designed  to  adapt  to  various  threats, 
rather  than  designed  to  address  a  specific  threat.  The  STANDARD  Missile 
program  has  been  developing  ship-based  air  defense  missiles  for  decades, 
so  there  was  an  established  program  office  that  invested  time  in  pre- 
Milestone-B  planning  and  coordination  with  stakeholders.  According  to 
program  officials,  the  original  plan  for  the  next  generation  missile  was  a 
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more  aggressive,  costly  solution  dubbed  SM-5.  After  thoroughly 
considering  alternatives,  however,  the  Navy  decided  to  take  the  more  cost 
conscious,  incremental  approach  of  the  SM-6  which  program  officials  said 
addressed  80  percent  of  their  capability  needs  for  half  the  cost  of  the  SM-5. 

Beyond  our  case  studies,  we  have  seen  other  successful  programs  in  the 
past,  including  the  F-16  Fighting  Falcon  fighter  aircraft  program,  that  also 
took  more  incremental  acquisition  approaches  based  on  well-defined 
requirements.  For  instance,  the  F-16  program  successfully  evolved 
capabilities  over  the  span  of  about  30  years.  Using  an  evolutionary 
approach  to  develop  the  aircraft  allowed  the  program  to  quickly  deliver 
new  and  improved  capabilities  to  the  warfighter,  and  to  increase  the 
aircraft’s  capability  as  new  technologies  were  matured  and  added  to  the 
aircraft.  The  first  increment,  developed  during  the  1970s,  provided  a  “day 
fighter”  aircraft  with  basic  air-to-air  and  air-to-ground  capabilities.  This 
allowed  the  developer  to  deliver  new  and  useful  military  capability  to  the 
warfighter  in  less  than  4  years.  With  each  subsequent  increment,  new 
technology  was  used  to  improve  the  engine,  radar,  structure,  avionics,  and 
other  systems  that  allow  the  aircraft  today  to  perform  close  air  support, 
ground  attack,  air  defense,  and  suppression  of  enemy  defense  missions. 
The  evolutionary  approach  also  enriched  the  industrial  base  capabilities 
by  extending  the  life  of  the  production  over  the  length  of  this  incremental 
approach. 

In  contrast,  we  have  previously  reported  on  many  acquisition  programs, 
including  the  Future  Combat  Systems  (FCS),  the  F-22  aircraft,  and  Joint 
Tactical  Radio  System,  that  have  proposed  unrealistic  and  poorly 
understood  requirements  and  pursued  revolutionary,  exotic  system 
solutions  which  were  ultimately  extremely  difficult  or  impossible  to 
achieve.  For  example,  the  FCS  program — which  was  comprised  of  14 
integrated  weapon  systems  and  an  advanced  information  network — was  to 
be  the  centerpiece  of  the  Army’s  effort  to  transition  to  a  lighter,  more  agile 
and  capable  combat  force.  However,  the  Army  started  this  ambitious 
program  in  May  2003  before  defining  what  the  systems  would  be  required 
to  do  and  how  they  would  interact.  It  did  not  expect  to  complete  defining 
requirements  until  at  least  2009,  6  years  after  program  initiation.  The 
program’s  failure  to  adequately  define  requirements  early  on  resulted  in 
design  changes  as  well  as  significant  cost  and  schedule  growth.  The  FCS 
program  has  recently  had  elements  cancelled  and  some  of  the  remaining 
elements  restructured  into  other  programs. 
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Stable  Programs  Leveraged 
Mature  Technologies  and 
Production  Techniques 


The  stable  programs  we  reviewed  also  leveraged  mature  technologies  and 
production  techniques,  anticipated  system  integration  challenges,  and 
demonstrated  the  feasibility  of  proposed  weapon  system  concepts  by 
completing  prototypes  before  awarding  development  contracts.  In  these 
programs,  the  technologies  typically  needed  to  meet  the  essential  system 
requirements  had  been  demonstrated  to  work  in  relevant  or  realistic 
environments.  Technologies  that  were  immature  generally  either  were  not 
considered  or  were  deferred  to  later  program  increments.  Some  of  the 
programs  also  used  established  production  lines,  fostering  stable 
production  capabilities  by  limiting  unknowns,  achieving  cost  efficiencies, 
and  accelerating  the  learning  process  necessary  to  develop  effective 
production  methods.  Furthermore,  these  programs  understood  the 
challenge  of  integrating  existing  technologies,  particularly  of  software 
items.  Three  of  our  five  case  study  programs — HIMARS,  JDAM,  and  SDB — 
developed  prototypes  before  Milestone  B,  which  allowed  for  the 
assessment  of  technologies,  ensured  that  integration  complexities  were 
understood,  and  demonstrated  the  feasibility  of  the  proposed  system 
concept.  For  example: 

•  HIMARS’s  system  feasibility  was  demonstrated  through  an  ACTD  project 
prior  to  Milestone  B.  According  to  an  early  program  manager,  although 
different  technological  solutions  were  utilized  during  formal  system 
development,  the  early  prototypes  developed  for  this  project  proved  that 
the  system  was  feasible.  In  addition,  HIMARS  leveraged  two  technologies 
already  in  production:  the  chassis  from  the  Family  of  Medium  Tactical 
Vehicles  program  and  the  rocket  pod  from  the  Multiple  Launch  Rocket 
System. 

•  The  P-8A  airframe  was  developed  from  the  Boeing  737  commercial 
aircraft,  avoiding  some  time  and  risk  inherent  in  developing  a  completely 
new  airframe.  In  addition,  according  to  program  officials,  the  P-8A 
airframe  is  being  produced  on  the  same  production  line  as  the  737  which 
provides  cost  efficiencies  and  a  decreased  production  learning  curve.  The 
program  did  not  have  fully  mature  technologies  at  development  start,  but 
identified  existing  technologies  as  back-ups.  P-8A  program  officials  also 
understood  that  software  integration  would  likely  be  more  complex  than 
the  contractor  predicted  and  allocated  resources  accordingly. 

•  Previous  iterations  of  the  STANDARD  Missile  allowed  significant  maturity 
in  the  SM-6  program  and,  therefore,  program  officials  said  they  focused 
development  on  integration  and  software  challenges.  SM-6  was  designed 
based  on  the  legacy  STANDARD  Missile  airframe  and  propulsion  systems 
and  the  Air  Force’s  Advanced  Medium  Range  Air-to-Air  Missile  active 
guidance  system — both  of  which  were  also  produced  by  Raytheon. 
According  to  program  officials,  this  has  allowed  the  program  to  use 
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Stable  Programs  Set  Realistic 
Cost  and  Schedule  Estimates 


existing  technologies  and  to  share  production  facilities,  which  has  in  turn 
produced  cost  efficiencies  for  the  program. 

SDB’s  acquisition  strategy  was  focused  from  the  beginning  on  utilization  of 
mature  technologies.  Program  officials  noted  that,  in  their  experience, 

SDB  had  an  unprecedented  level  of  design  maturity  and  production 
readiness  prior  to  Milestone  B — the  SDB  guidance  system,  warhead,  and 
link  kit  were  all  developed  prior  to  program  start.  According  to  the  Air 
Force’s  Deputy  for  Acquisition,  SDB  developed  competitive  prototypes, 
giving  equal  funding  to  two  contractors  with  the  goal  of  demonstrating 
maturity  of  their  concepts  prior  to  Milestone  B.  During  source  selection, 
contractors  were  only  given  credit  for  demonstrated  performance  of  their 
prototypes — not  for  performance  promised  for  the  future.  Program 
officials  told  us  that  this  meant  that  the  program  entered  the  system 
development  phase  with  production  representative  hardware  that  met  the 
requirements — building  more  units,  readying  the  factory  for  low  rate 
production,  and  completing  tests  were  the  only  jobs  remaining  for  the  then 
sole-source  contractor.  The  program  also  conducted  a  critical  design 
review  with  each  contractor  prior  to  Milestone  B. 

Program  officials  told  us  that  technologies  for  JDAM  tail-kits  were 
developed  and  demonstrated  in  a  research  environment  several  years 
before  the  program  began.  Similar  to  SDB,  program  officials  told  us  that 
JDAM  utilized  competitive  prototyping  to  develop  and  test  proposed 
technologies.  This  allowed  the  program  to  incentivize  contractors  to 
achieve  a  prototype  with  a  low  unit  cost  prior  to  awarding  the  system 
development  contract. 

In  contrast,  we  have  previously  reported  that  many  programs  rely  on 
immature  technologies  and  do  not  invest  in  prototypes  before  starting 
development.  For  instance,  the  Armed  Reconnaissance  Helicopter  (ARH) 
failed  in  large  part  due  to  misunderstanding  the  level  of  development 
required  to  integrate  a  commercial  solution.  In  addition,  the  F-22  was 
based  on  advanced  technologies  and  experienced  considerable  problems 
during  development  due  to  a  lack  of  an  existing  industrial  and  supplier 
base  experienced  in  working  with  one  another  in  fabricating,  assembling 
and  producing  the  high  technology  components  necessary  for  the  aircraft. 

It  is  well  recognized  that  realistic  cost  and  schedule  estimates  that  account 
for  program  risks  are  imperative  to  establishing  a  sound  basis  for 
acquiring  new  weapon  systems.  The  foundation  of  a  realistic  estimate  is  a 
high  degree  of  knowledge  about  program  requirements,  technology, 
design,  and  manufacturing.  The  stable  programs  we  reviewed  had  realistic 
cost  and  schedule  estimates  at  Milestone  B  because  they  had  a  good 
understanding  of  what  was  needed  to  develop  and  produce  the  proposed 
systems.  Since  Milestone  B,  our  case  study  programs  have  generally 
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tracked  to  their  initial  development  funding  profiles.  Specific  examples  of 
good  cost  estimating  and  risk  analysis  from  our  case  studies  include 

•  The  P-8A  program  was  funded  to  an  independent  cost  estimate  which  was 
about  14  percent  higher  than  the  service  cost  estimate.  According  to 
program  officials,  the  independent  cost  estimators  deemed  the  program’s 
estimate  for  software  development  to  be  insufficient,  and  included 
additional  funding  in  the  cost  estimate  for  this  effort.  After  contract  award, 
the  Navy  added  $500  million  to  the  contract  to  ensure  adequate  early  effort 
for  software  development.  The  December  2007  development  cost  estimate 
was  about  4  percent  lower  than  the  original  2004  estimate.  (See  fig.  5.) 


Figure  5:  Comparison  of  Original  and  2007  Development  Funding  Estimates  for 
P-8A 

Fiscal  year  2004  dollars  in  millions 
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-  6/2004  SAR 
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Source:  DOD  reported  2004  and  2007  SAR  data. 


•  The  SM-6  program  also  effectively  estimated  costs — according  to  an  early 
program  manager,  the  program  insisted  on  including  all  related  costs  in  its 
estimate,  including  field  activity  costs  and  the  program’s  share  of 
department-wide  overhead  and  expenses.  The  program  also  allocated  risk 
across  the  whole  program,  building  in  margin  for  each  step.  Because  the 
program  made  a  point  to  develop  doable  requirements,  it  had  the 
prerequisite  knowledge  about  technologies  and  design  to  make  an 
accurate  estimate  and  establish  a  realistic  funding  profile.  (See  fig.  6.) 
According  to  an  early  program  manager,  to  realistically  estimate  schedule 
the  program  conducted  a  comparative  study  of  major  missile  development 
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programs.  From  this  study  they  concluded  that  all  of  these  programs  have 
taken  between  9  and  12  years  to  get  from  Milestone  B  to  initial  capability. 
They  used  these  historical  numbers  as  the  basis  for  the  SM-6  schedule 
estimate. 


Figure  6:  Comparison  of  Original  and  2007  Development  Funding  Estimates  for 
SM-6 

Fiscal  year  2004  dollars  in  millions 
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Source:  DOD  reported  2004  and  2007  SAR  data. 


Some  of  the  stable  programs  we  reviewed  also  included  additional  margin 
upfront  for  risky  activities  when  constructing  schedule  estimates.  A 
former  P-8A  program  official  stated  that  the  program’s  development 
contract  included  12  months  of  margin  for  risk;  and  a  former  SM-6  official 
said  the  program  specifically  included  an  assumption  of  two  flight  failures 
during  testing — each  entailing  2  to  3  months’  time  to  address — explaining 
that  no  program  realistically  goes  through  all  of  its  tests  without  a  failure. 
In  addition,  this  official  told  us  that,  instead  of  trying  to  eliminate  risk  by 
simply  adding  on  a  single  schedule  “cushion”  at  the  end  of  the 
development  cycle,  they  allocated  schedule  risk  margin  more  strategically. 
First,  the  program  identified  the  specific  tasks  that  were  sources  of  the 
greatest  schedule  risk  and  then  they  added  a  margin  to  the  scheduled  time 
allocated  for  each  task.  This  decreased  the  chance  that  contractor 
personnel  would  be  brought  on  board  (and  paid)  to  work  on  any  specific 
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task  prematurely,  which  in  turn  helped  to  decrease  the  cost  consequences 
of  schedule  delays. 

In  contrast,  others  who  began  with  overly  optimistic  cost  and  funding 
assumptions  have  required  much  more  funding  per  year  than  first 
requested.  For  example,  the  Armed  Reconnaissance  Helicopter  (ARH) 
program  rushed  through  the  planning  process,  skipping  key  systems 
engineering  steps  in  a  drive  to  obligate  remaining  funding  from  its 
predecessor  program,  the  terminated  Comanche  reconnaissance 
helicopter.  In  2009,  we  found  that  the  analysis  of  alternatives  for  ARH 
looked  at  only  two  options — improvement  of  the  existing  system  or 
procurement  of  nondevelopmental  helicopters.  We  also  found  that  the 
program  did  not  adequately  assess  risk  for  these  two  alternatives. 11 
According  to  program  officials,  the  plan  the  program  chose  did  not  have 
room  to  trade-off  cost,  schedule,  or  performance.  Schedule  estimates  were 
driven  by  a  desired  fielding  date  and  cost  was  determined  primarily  by 
multiplying  the  desired  unit  cost  by  the  number  of  desired  aircraft.  These 
cost  and  schedule  requirements — which  program  officials  said  were 
directed  by  Army  leadership — were  developed  without  an  understanding 
of  the  issues  or  a  thorough  vetting  with  relevant  industry  stakeholders.  As 
a  result,  within  2  years  of  Milestone  B,  actual  and  estimated  development 
costs  had  quickly  escalated  and  the  development  schedule  had  been 
extended.  Ultimately,  it  was  determined  that  the  strategy  was  not 
executable  and  the  program  was  terminated  in  October  2008.  Figure  7 
shows  the  program’s  original  funding  estimate  from  2005,  based  on  a  lack 
of  knowledge  about  the  weapon  system’s  requirements  and  the  resources 
it  would  take  to  deliver  it,  and  its  funding  estimate  prior  to  termination. 

We  note  that  the  estimated  development  funds  required  more  than 
doubled — an  increase  of  almost  $365  million  (fiscal  year  2005  dollars) — 
and  the  development  cycle  time  increased  by  27  months  between  the 
program’s  start  to  December  2007. 


uGAO,  Defense  Acquisitions:  Many  Analyses  of  Alternatives  Have  Not  Provided  a  Robust 
Assessment  of  Weapon  System  Options,  GAO-09-665  (Washington,  D.C.:  September  24, 
2009). 
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Figure  7:  Comparison  of  Original  and  2007  Development  Funding  Estimates  for  the 
ARH 

Fiscal  year  2005  dollars  in  millions 
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180 
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-  7/2005  SAR 

-  12/2007  SAR 

Source:  DOD  reported  2005  and  2007  SAR  data. 


The  Air  Force’s  Global  Hawk  unmanned  aerial  vehicle  was  approved  for  a 
simultaneous  development  start  and  low  rate  production  in  2001.  DOD 
restructured  the  Global  Hawk  acquisition  strategy  in  March  2002  to 
include  a  second  Global  Hawk  model.  The  program  office  estimated  the 
cost,  time,  and  funding  required  to  develop  this  new,  bigger  model  with 
more  stringent  requirements  without  a  suffiicient  understanding  of  the 
new  model.  It  then  restructured  the  program  again  in  December  2002; 
changing  the  capabilities  required  in  the  new  variant.  As  a  result,  the  initial 
cost  estimates  had  become  outdated  within  2  years  of  development  start, 
and  estimated  total  development  costs  had  more  than  doubled.  Figure  8 
shows  the  program’s  original  funding  estimate  from  2001  and  its  funding 
estimate  as  of  December  2007.  Estimated  development  funding  has  more 
than  tripled  and  the  development  program  has  been  extended  by  about  7 
years. 


Page  24 


GAO-10-522  Defense  Acquisitions 


Once  Under  Way,  Programs 
Resisted  Adding  New 
Requirements 


Figure  8:  Comparison  of  Original  and  2007  Development  Funding  Estimates  for 
Global  Hawk 

Fiscal  year  2000  dollars  in  millions 
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Source:  DOD  reported  2001  and  2007  SAR  data. 


After  starting  development,  our  case  study  programs  resisted  adding  new 
requirements  by  keeping  stakeholders  focused  on  the  importance  of 
adhering  to  cost  and  schedule,  as  well  as  performance  commitments.  For 
example,  P-8A  officials  related  that  one  reason  for  cost  and  schedule 
stability  was  the  program  office’s  willingness  to  limit  capability  and 
requirements  changes  proposed  by  the  P-3C  user  community,  and  in 
particular,  to  find  workable  solutions  to  user-preferred  “gadgets.”  For 
instance,  to  detect  submarines,  the  P-3C  used  a  technology  that  measured 
shifts  in  the  earth’s  magnetic  field.  The  user  community  insisted  that  P-8A 
use  this  technology,  even  though  contractor  engineers  determined  that  it 
would  require  the  development  of  veiy  expensive  software  to  account  for 
differences  in  the  airframes’  structures.  The  P-8A  program  office 
successfully  worked  with  the  user  community  to  gain  acceptance  of  an 
alternative  technology  which  provided  the  same  submarine  detection 
capability  while  keeping  the  program  on  cost. 

In  the  JDAM  program,  some  new  requirements  were  added  but  according 
to  program  officials  they  did  not  alter  the  basic  performance  parameters  of 
the  program.  One  JDAM  capability  enhancement — a  laser  sensor — was 
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Programs  Maintained  Stable 
Funding  During  Execution 


developed  by  the  contractor  with  its  own  resources.  In  contrast,  unstable 
programs  sometimes  chase  performance,  with  less  concern  for  cost 
implications.  Global  Hawk,  for  example,  added  new,  major,  and  unplanned 
requirements  after  Milestone  B,  increasing  cost  and  schedule  significantly. 
Rather  than  establishing  separate  program  increments,  Global  Hawk 
restructured  its  original  program  to  add  a  new  aircraft  variant  with 
enhanced  capabilities. 

In  addition  to  starting  with  annual  funding  baselines  based  on  realistic 
cost  estimates,  as  discussed  above,  the  stable  programs  we  reviewed 
typically  received  annual  development  appropriations  close  to  their  full 
funding  requests.  For  example,  the  P-8A  program  received  96  percent  of  its 
requested  development  funding  for  the  years  2005  to  2007.  (See  fig.  9.) 


Figure  9:  P-8A  Annual  Development  Funding  Requested  and  Received 


Dollars  in  then-year  millions 
1,200 


Fiscal  year 

Requested 

Received 

Source:  Navy  budget  justification  documents. 

Although  these  data  show  the  amount  received  in  each  year  was  slightly 
less  than  requested,  P-8A  program  officials  stated  that  overall  the  program 
has  had  very  stable  funding.  They  attributed  the  stability  of  funding  to 
factors  including  (1)  the  acute  need  for  the  P-8A  to  be  procured  quickly 
and  (2)  the  steady  development  of  the  aircraft  accompanied  by  remaining 
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on  track  with  initial  cost  and  schedule  goals  from  the  outset  of  the 
program. 

Officials  from  the  other  stable  programs  echoed  these  themes — 
particularly  that  good  program  execution  inspired  confidence  in  the 
program — when  asked  to  explain  relatively  stable  funding.  In  addition, 
HIMARS  program  officials  told  us  they  maintained  good  communication 
with  their  liaison  to  the  funding  community,  ensuring  that  this  liaison  was 
fully  informed  of  the  status  and  progress  of  the  program.  They  felt  that  this 
was  extremely  important,  because  a  thorough  and  current  knowledge  of 
the  program  is  what  allows  a  liaison  to  advocate  for  the  program  and 
protect  against  cuts  in  funding.  In  addition,  officials  from  the  SM-6 
program  discussed  proactively  anticipating  and  responding  to  funding 
cuts.  Program  officials  track  what  kind  of  funding  cuts  they  could  handle 
and  what  the  effects  might  be.  Program  officials  stated  that  when  there  is  a 
request  to  take  funding  from  the  program,  they  always  take  the 
opportunity  to  respond  and  justify  why  the  program  cannot  spare  the 
money.  Often  that  justification  is  accepted. 

Funding  stability  is  an  essential  ingredient  to  a  successful  program. 
However,  in  view  of  the  many  pressures  that  characterize  the  acquisition 
culture,  stable  funding  and  support  alone  will  not  prevent  other  acquisition 
problems,  such  as  problems  stemming  from  unrealistic  performance 
requirements,  immature  technologies,  and  highly  concurrent  schedules. 
Funding  instability  has  often  been  pointed  to  by  program  managers  as  a 
factor  contributing  to  program  instability.  Given  that  there  are  too  many 
programs  for  available  resources  and  many  programs  encounter  cost, 
schedule,  and  performance  problems,  it  is  not  unexpected  that  some 
programs  experience  funding  instability.  However,  we  have  also  seen  that 
funding  instability  can  be  the  result,  not  the  cause,  of  performance 
problems.  For  example,  in  2002,  we  found  that,  while  the  F-22  program 
office  attributed  some  of  its  production  cost  increases  to  a  reduction  in 
quantities,  the  program  had  been  significantly  affected  by  design  and 
manufacturing  problems  that  started  during  development. 12  For  example, 
in  1997,  an  independent  review  team  determined  that  the  product 
development  effort  was  underestimated.  In  short,  successful  programs 
enjoy  funding  stability,  but  funding  stability  does  not  ensure  program 
success. 


12GAO,  Best  Practices:  Capturing  Design  and  Manufacturing  Knowledge  Early  Improves 
Acquisition  Outcomes,  GAO-02-701  (Washington,  D.C.:  July  15,  2002). 
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Recent  Policy  and 
Legislative  Reform 
Initiatives  Reflect  Key 
Characteristics  of 
Successful  Programs 


Recently,  Congress  and  DOD  have  taken  major  steps  towards  reforming 
the  defense  acquisition  system  in  ways  that  may  increase  the  likelihood 
that  weapon  programs  succeed  in  meeting  planned  cost  and  schedule 
objectives.13  Many  of  these  steps  are  consistent  with  key  elements  we 
found  in  our  five  stable  case  study  programs.  In  particular,  the  new  DOD 
policy  and  legislative  provisions  place  greater  emphasis  on  front-end 
planning  and  establishing  sound  business  cases  for  starting  programs.  For 
example,  the  provisions  strengthen  systems  engineering  and  cost 
estimating,  and  require  early  milestone  reviews,  prototyping,  and 
preliminary  designs.  They  are  intended  to  enable  programs  to  refine  a 
weapon  system  concept  and  make  cost,  schedule,  and  performance  trade¬ 
offs  before  significant  commitments  are  made.  Fundamentally,  the 
provisions  should  help  programs  replace  risk  with  knowledge,  and  set  up 
more  executable  programs.  Key  DOD  and  legislative  provisions  compared 
with  factors  we  identified  in  stable  programs  are  summarized  in  table  3. 


Table  3:  Comparison  of  Factors  Contributing  to  Stable  Programs  and  Recent 
Acquisition  Reform  Initiatives 

Stability  factors 

Recent  acquisition  reform  initiatives 

Establish  a  sound,  executable 
business  case 

Overall,  strong  emphasis  on  front-end  planning  (pre¬ 
systems  acquisition) 

•  incremental  approach  to 
acquiring  capabilities 

•  incremental  development  emphasized,  with 
each  increment  that  provides  a  significant 
increase  in  capability  to  be  managed  separately 

•  clear,  well  defined  •  early  reviews  to  be  conducted  prior  to  start  of 


requirements  development  (Milestone  B) 


•  enhanced  requirements  for  Analysis  of 
Alternatives 


•  new  leadership  positions  established  to 
enhance  systems  engineering  and 
developmental  testing 

•  leverage  mature  technologies  •  independent  review  of  technology  maturity  and 

integration  risk  prior  to  Milestone  B 

•  competitive  prototypes 

•  Preliminary  Design  Review  to  be  conducted 
earlier,  prior  to  Milestone  B 


13The  Weapon  Systems  Acquisition  Reform  Act  of  2009,  Pub.  L.  No.  111-23,  was  enacted 
May  22,  2009.  In  December  2008,  DOD  revised  its  acquisition  instruction — Department  of 
Defense  Instruction  5000.02,  Operation  of  the  Defense  Acquisition  System. 
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Stability  factors 

Recent  acquisition  reform  initiatives 

•  establish  realistic  cost  and 
schedule  estimates 

•  new  position  and  organization  established  to 
review  and  conduct  independent  cost  estimates 
for  MDAPs  and  provide  cost  estimating 
guidance  DOD-wide 

•  early  cost  estimate  required  for  M  ilestone  A 

•  confidence  level  for  cost  estimates  to  be 
reported 

Execute  business  case  in 
disciplined  manner 

•  resist  new  requirements 

•  configuration  steering  boards  established  to 
stabilize  requirements 

•  post-Critical  Design  Review  assessment 
required  to  review  progress 

•  stable  funding 

Source:  GAO  analysis  of  the  Weapon  Systems  Acquisition  Reform  Act  of  2009,  Pub.  L.  No.  111-23,  and  Department  of  Defense 
Instruction  5000.02  (December  8,  2008). 


While  it  is  too  soon  to  determine  if  Congress  and  DOD’s  reform  efforts  will 
improve  weapon  program  outcomes,  we  have  seen  evidence  that  DOD  is 
taking  steps  to  implement  the  provisions.  For  example,  in  December  2009, 
the  department  issued  a  new  implementation  policy,  which  identifies  roles 
and  responsibilities  and  institutionalizes  many  of  the  requirements  of  the 
Weapon  Systems  Acquisition  Reform  Act  of  2009.  It  has  also  filled  several 
key  leadership  positions  created  by  the  legislation,  including  the  Directors 
for  Cost  Analysis  and  Program  Evaluation,  Developmental  Test  & 
Evaluation,  and  Performance  Assessments  and  Root  Cause  Analyses.  To 
increase  oversight,  the  department  has  embarked  on  a  5-year  effort  to 
increase  the  size  of  the  acquisition  workforce  by  up  to  20,000  personnel  in 
2015.  Furthermore,  the  department  has  begun  applying  the  acquisition 
reform  provisions  to  some  new  programs  currently  in  the  planning 
pipeline.  For  example,  many  of  the  pre-Milestone-B  programs  we  reviewed 
this  year  as  part  of  our  annual  assessment  of  selected  weapon  programs 
plan  to  develop  competitive  prototypes  and  conduct  a  preliminary  design 
reviews  before  going  to  Milestone  B.14  In  the  Joint  Air-to-Ground  Missile 
program,  the  Army  recently  awarded  two  contracts  for  a  27-month 
technology  development  phase  which  will  culminate  in  test  flights  of 
competing  prototypes  prior  to  Milestone  B. 

The  success  of  DOD’s  efforts,  however,  will  depend  in  part  on  how 
consistently  the  new  provisions  are  implemented  and  reflected  in 


14GAO-10-388SP. 
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decisions  on  individual  programs.  In  the  past,  inconsistent  implementation 
of  existing  policy  hindered  DOD’s  ability  to  plan  and  execute  programs 
effectively.  Inconsistent  implementation  occurred  in  part  because  decision 
makers  were  not  held  accountable  for  programs  outcomes  and  there  were 
few,  if  any,  consequences  when  programs  ran  into  problems.  Furthermore, 
cultural  and  environmental  forces  at  DOD  work  against  sound 
management  practices.  These  forces  encourage  programs  to  pursue  overly 
ambitious  requirements  and  lengthy  development  efforts,  and  to  move 
forward  with  risky  and  unexecutable  acquisition  strategies.  We  have  found 
too  often  that  program  sponsors  overpromise  capabilities  and 
underestimate  costs  in  order  to  capture  the  funding  needed  to  start  and 
sustain  development  programs.  For  acquisition  reforms  to  be  effective, 
they  will  have  to  address  these  forces  as  well.  For  example,  while 
acquisition  reform  provisions  are  intended  to  make  cost  estimates  more 
reliable  and  realistic,  the  provisions  may  be  compromised  by  the 
competition  for  funding  that  encourages  programs  to  appear  affordable 
when  they  are  not.  Furthermore,  when  program  sponsors  present  a 
program  as  more  than  a  weapon  system,  but  rather  as  essential  to  new 
fighting  concepts,  pressures  exist  to  accept  less-than-rigorous  cost 
estimates.  If  reform  is  to  succeed,  then  programs  that  present  realistic 
strategies  and  resource  estimates  must  succeed  in  winning  approval  and 
funding,  as  was  the  case  with  the  stable  programs  we  reviewed.  Those 
programs  that  continue  past  practices  of  pushing  unexecutable  business 
cases  must  be  denied  funding  before  they  begin. 

DOD  will  also  need  to  ensure  that  adequate  resources — funding  and 
workforce  capacity — are  available  to  support  the  front-end  planning 
activities  now  required  for  new  weapon  programs.  In  the  past,  budget 
realities  within  DOD  have  made  it  more  advantageous  to  fund  technology 
development  in  acquisition  programs  because  most  of  the  department’s 
research  and  development  funding  goes  to  existing  programs  of  record. 
Weapon  system  programs  have  historically  received  about  80  percent  of 
the  department’s  research  and  development  budget  whereas  science  and 
technology  activities  have  received  about  20  percent  of  the  budget.  The 
money  going  toward  science  and  technology  is  spread  over  several 
thousand  projects,  while  the  money  going  toward  weapon  programs  is 
spread  out  over  considerably  fewer  projects.  This  “distribution  of  wealth” 
makes  it  easier  to  finance  technology  development  within  an  acquisition 
program.  Once  initiated,  a  program  is  in  a  more  competitive  position  to 
attract  funding  support  from  within  the  department.  With  competition  for 
funding  intense,  due  to  the  high  demand  for  weapon  systems  and  other 
needs  in  the  department,  freeing  up  funds  for  pre-systems  acquisition 
activities  may  be  challenging.  Yet,  as  we  have  seen  in  stable  programs, 
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strong  leadership  support  ensured  that  there  was  sufficient  funding  and 
other  resources  to  effectively  plan  up  front. 

Because  programs  in  the  past  tended  to  proceed  to  Milestone  B  too 
quickly,  there  may  be  limited  experience  in  DOD  in  conducting  many  of 
the  program  planning  activities  now  required  by  the  acquisition  reform 
initiatives.  Lessons  learned  from  stable  programs,  such  as  the  case  study 
programs  we  reviewed,  could  serve  as  useful  tools  for  the  successful 
implementation  of  the  reform  initiatives.  For  example,  these  programs  had 
the  following:  effective  and  consistent  leadership  in  place  early  on; 
program  managers  who  were  empowered  to  plan  and  establish  a  sound 
business  case  for  starting  a  program;  resources  available  for  conducting 
front-end  systems  engineering  and  planning;  established  mechanisms  to 
engage  industry  early  on  and  to  contract  for  prototypes;  flexibility  to  make 
cost,  schedule  and  performance  trade-offs  before  committing  to  a  business 
case;  cost  and  schedule  baselines  that  realistically  accounted  for  risks;  and 
evolutionary  acquisition  approaches  that  address  capability  needs  in 
achievable  increments  based  on  well-defined  requirements. 


Conclusions 


Although  most  DOD  programs  fail  to  meet  their  intended  cost  and 
schedule  objectives,  some  programs  have  still  been  successful.  No  one 
factor  will  ensure  success;  instead,  a  lot  of  things  need  to  go  right  in  both 
planning  and  implementing  a  program.  However,  it  is  critical  to  get  the 
systems  engineering  and  planning  phase  right,  because  extraordinary 
implementation  cannot  save  a  program  with  a  business  case  that  was 
flawed  from  the  beginning.  In  our  case  studies,  we  found  that  stable 
programs  established  sound,  knowledge-based  business  cases  before 
moving  forward  and  then  executed  them  in  a  disciplined  manner.  How 
they  were  able  to  successfully  do  this  was  largely  due  to  strong  leadership 
support  and  proactive  program  managers  who  knew  how  to  get  results. 
Getting  the  right  people  in  place  at  the  right  time  and  supporting  them 
with  the  requisite  resources  is  critical.  However,  programs  are  also  more 
likely  to  succeed  when  there  is  a  sense  of  urgency  to  deliver  a  needed 
capability  and  senior  leadership  views  the  program  as  a  high  priority.  In 
addition,  programs  benefit  from  having  experienced  program  managers 
who  provide  consistent  leadership  through  major  phases  of  a  program. 

Relying  on  strong  leadership  or  treating  each  program  as  a  priority  is  not 
scalable  across  DOD’s  broad  portfolio  of  weapon  programs.  Rather,  good 
program  outcomes  ought  to  occur  normally  as  an  outgrowth  of  effective 
policy,  processes,  priorities,  oversight,  and  leadership.  Acquisition  policy 
and  processes  establish  rules  and  mechanisms  that  facilitate  good 
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program  decisions.  However,  rules  and  mechanisms  are  only  as  good  as 
the  people  that  implement  them.  The  new  round  of  DOD  and 
Congressional  acquisition  reforms  create  a  renewed  opportunity  to 
improve  acquisition  outcomes,  but  only  if  it  is  accompanied  with  the 
appropriate  leadership  support  that  allowed  the  stable  programs  we 
reviewed  to  establish  reasonable  business  cases  and  execute  them  with 
confidence.  If  reform  is  to  succeed,  programs  that  present  realistic 
strategies  and  resource  estimates  must  succeed  in  winning  approval  and 
funding.  Those  programs  that  continue  past  practices  of  pushing 
unexecutable  strategies  must  be  denied  funding  before  they  begin. 


Agency  Comments 
and  Our  Evaluation 


In  written  comments  on  a  draft  of  this  report,  DOD  stated  that  it  was 
encouraged  that  the  report  cites  progress  made  over  the  past  several  years 
to  improve  acquisition  processes  and  reduce  cost  and  schedule  growth. 
DOD’s  response  is  reprinted  in  appendix  II.  DOD  noted  that  it  has  recently 
instituted  several  major  changes  to  acquisition  policy  that  are  aimed  at 
starting  programs  right  by  using  early  planning  and  systems  engineering, 
joint  analysis  teams,  competitive  prototyping,  configuration  steering 
boards,  credible  cost  estimates,  and  program  manager  service  agreements. 
DOD  anticipates  improvements  in  program  performance  in  the  ensuing 
years  due  to  the  recent  acquisition  reform  initiatives. 

DOD  agreed  with  the  reasons  we  found  for  program  success,  such  as 
strong  leadership,  disciplined  program  managers,  executable  business 
cases,  and  achievable  increments  based  on  well-defined  requirements. 
However,  DOD  pointed  out  that  our  findings  are  based  upon  small,  less 
costly,  and  less  complicated  programs  which  may  not  be  readily  scalable 
to  large,  software  intensive  systems  such  as  satellites  or  the  Joint  Strike 
Fighter.  While  we  agree  that  more  complex  weapon  system  programs 
present  greater  challenges,  we  have  previously  reported  that  DOD  should 
consider  increasing  the  number  of  programs  which  provide  incremental 
improvements  in  capability  to  the  warfighter  in  a  timely  way.  Complex 
programs  that  we  cite  in  our  report,  such  as  the  F/A-18E/F  and  the  F-16, 
were  able  to  balance  requirements  with  available  resources  and  produce 
cutting-edge  weapon  systems  within  cost  and  schedule  targets. 
Revolutionary  efforts  that  rely  on  unproven  technological  breakthroughs 
should  be  the  exception  rather  than  the  rule.  In  addition,  regardless  of 
program  complexity,  the  decision  to  begin  a  weapon  program  should  be 
knowledge-based  and  decision  makers  need  to  be  fully  informed  of  the 
risks  and  scope  of  the  effort  to  be  undertaken.  In  the  past,  the  decisions  to 
enter  into  revolutionary  acquisition  programs  such  as  the  Joint  Strike 
Fighter  were  based  on  overly  optimistic  assumptions  of  the  cost  and  time 
involved  to  acquire  these  systems. 
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DOD  also  noted  that  although  the  measures  we  used  to  measure  program 
performance  are  useful — change  in  development  cost,  unit  cost,  and 
schedule  from  the  original  program  baseline — they  are  only  valid  when 
operational  requirements  remain  static  throughout  the  program  lifecycle. 
According  to  DOD,  in  some  cases,  the  warfighter’s  needs  change  and  the 
department  must  adapt  by  enhancing  weapon  system  functionality.  We 
agree  that  the  needs  of  the  warfighter  are  paramount;  however,  DOD 
would  be  in  a  better  position  to  adapt  to  the  changing  needs  of  the 
warfighter  by  reducing  the  time  it  takes  to  field  new  systems  and 
enhancing  weapon  system  functionality  in  future  increments.  According  to 
DOD  acquisition  policy,  incremental  development  is  the  preferred 
approach  and  each  increment  that  provides  a  significant  increase  in 
capability  should  be  managed  separately. 

DOD  agreed  that  program  manager  tenure  is  a  contributing  factor  in 
program  stability,  but  thought  that  our  characterization  of  tenure  as  an 
average  is  misleading  because  it  does  not  reflect  the  total  time  program 
managers  serve  in  their  positions.  The  baseline  average  we  calculated  is 
based  on  data  collected  as  part  of  our  2007  report  on  program  manager 
empowerment  and  accountability. 16  Our  work  has  shown  that  rather  than 
having  lengthy  assignment  periods  between  key  milestones  as 
suggested  by  best  practices,  many  programs  we  have  reviewed  had 
multiple  program  managers  within  the  same  milestone.  Furthermore, 
the  key  point  we  are  making  in  the  report  is  that  program  manager  tenure 
in  our  case  study  programs,  calculated  using  the  same  methodology  used 
in  our  previous  report,  was  longer  than  what  we  have  seen  in  other 
programs  (2.4  versus  1.4  years),  which  was  a  contributing  factor  to  their 
relative  success. 

DOD  also  questioned  our  criteria  for  defining  “stable”  and  “unstable” 
programs  and  thought  that  our  evaluation  criteria  were  subjective  and 
should  have  been  based  on  accepted  standards,  legislation,  and 
regulations.  However,  aside  from  the  Nunn-McCurdy  breach  criteria 
established  by  Congress  for  “significant”  and  “critical”  unit  cost  growth  in 
major  weapon  programs — 30  and  50  percent  growth  respectively  from  the 
original  baseline — there  are  no  standard  criteria  established  for  assessing 
weapon  program  stability.  We  are  not  suggesting  that  the  criteria  used  in 
our  report  is  the  only  way  to  measure  program  stability;  however,  we 
believe  it  is  important  to  examine  programs  from  several  perspectives — 
growth  in  development  and  unit  costs,  and  delay  in  achieving  initial 


1EGAO-08-62R. 
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operational  capability.  We  selected  thresholds  for  these  three  indicators 
based  on  historical  cost  and  schedule  growth  in  major  defense  acquisition 
programs  and  our  judgment  based  on  many  years  of  conducting  DOD 
program  reviews. 

DOD  also  provided  technical  comments  which  we  incorporated  where 
appropriate. 


We  are  sending  copies  of  this  report  to  the  Secretary  of  Defense  and 
interested  congressional  committees.  In  addition,  this  report  will  be  made 
available  at  no  charge  on  the  GAO  Web  site  at  http://www.gao.gov.  If  you 
or  your  staff  have  any  questions  about  this  report  or  need  additional 
information,  please  contact  me  at  (202)  512-4841  or  sullivanm@gao.gov. 
Contact  points  for  our  Offices  of  Congressional  Relations  and  Public 
Affairs  may  be  found  on  the  last  page  of  this  report.  GAO  staff  who  made 
major  contributions  to  this  report  are  listed  in  appendix  III. 


Michael  J.  Sullivan,  Director 
Acquisition  and  Sourcing  Management 
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This  report  identifies  and  examines  some  major  defense  acquisition 
programs  that  have  stayed  relatively  close  to  the  cost  and  schedule 
estimates  established  when  they  started  development,  in  order  to  identify 
useful  lessons  that  can  be  implemented  with  the  current  acquisition 
reforms.  Specifically,  our  objectives  were  to  (1)  identify  and  describe 
programs  within  the  Department  of  Defense’s  (DOD)  2008  major  weapon 
system  acquisition  program  portfolio  that  were  stable  and  on  track  to  meet 
cost  and  schedule  targets  outlined  at  program  development  start;  (2) 
determine  what  factors  enabled  some  stable  programs  to  achieve  these 
cost  and  schedule  targets;  and  (3)  analyze  recent  acquisition  reform 
initiatives  to  determine  how  lessons  learned  from  these  stable  programs 
can  be  of  use  as  DOD  implements  acquisition  reform. 

To  identify  and  describe  programs  that  were  stable,  we  analyzed  data  from 
DOD’s  Selected  Acquisition  Reports  (SAR),  as  well  as  other  program  data. 
DOD  typically  submits  SARs  on  current  major  defense  acquisition 
programs  to  Congress  at  the  end  of  the  first  quarter  of  each  fiscal  year, 
which  provide  a  basis  to  determine  each  program’s  cost  and  schedule 
performance.  We  first  obtained  the  list  of  the  programs  that  published 
SARs  in  December  2007  (the  last  year  in  which  full  SARs  were  published, 
as  of  the  time  we  conducted  our  work)  from  the  December  2007  SAR 
summaiy  tables  posted  on  DOD’s  public  acquisition  Web  site.  We  excluded 
programs  for  which  December  2007  was  the  first  SAR  or  the  termination 
SAR,  as  we  could  not  make  valid  baseline  comparisons  for  these 
programs.  We  also  made  other  minor  adjustments  to  the  program  list  to 
enable  valid  baseline  comparisons,  such  as  separating  subprograms  that 
were  listed  as  one  in  the  2007  SAR  table  but  reported  the  relevant  data 
separately.  We  were  left  with  a  list  of  95  programs,  for  which  we  obtained 
SAR  data  and  other  information  through  the  Defense  Acquisition 
Management  Information  Retrieval  (DAMIR)  Purview  system. 1  We  also 
analyzed  data  submitted  to  us  by  program  offices  as  part  of  our  annual 
review  of  selected  weapon  systems.  We  then  excluded  from  our  analysis 
programs  for  which  not  all  data  necessary  for  our  baseline  comparisons 


'DAMIR  Purview  is  an  executive  information  system  operated  by  the  Office  of  the  Under 
Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  /  Acquisition  Resources  and 
Analysis. 
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were  published,  as  well  as  those  which  passed  Milestone  B  less  than  3 
years  prior  to  the  December  2007  SAR  report.2 

We  analyzed  each  of  the  remaining  63  programs  based  on  data  we 
obtained  from  DAMIR  and  from  program  offices.  We  retrieved  data  from 
each  program’s  original  development  baseline  report  that  showed 
estimated  total  amount  of  research,  development,  test,  and  evaluation 
(RDT&E)  costs,  total  program  acquisition  unit  costs  (PAUC),  and  date  of 
initial  operational  capability  for  the  program,  as  of  the  start  of 
development.  We  then  obtained  comparable  data  on  each  program  from  its 
December  2007  SAR.  We  converted  all  cost  information  to  fiscal  year  2009 
dollars  using  conversion  factors  from  the  DOD  Comptroller’s  National 
Defense  Budget  Estimates  for  Fiscal  Year  2009  (Table  5-9).  Through 
discussions  with  DOD  officials  responsible  for  the  DAMIR  database  and 
confirming  selected  data  with  program  offices,  we  had  previously 
determined  that  the  SAR  data  and  the  information  retrieved  from  DAMIR 
were  sufficiently  reliable  for  our  purposes. 

Using  these  data,  we  assessed  program  stability  for  each  program  as 
described  below.  We  defined  “program  stability”  to  mean  minimal  change 
from  first  full  cost  and  schedule  estimates  to  the  December  2007 
estimates.  Because  it  was  not  feasible  to  compare  original  baseline  versus 
current  program  performance  on  all  relevant  parameters,  we  chose  three 
that  best  provided  an  indication  of  whether  a  program  had  remained  close 
to  its  cost  and  schedule  estimates.  These  three  stability  indicators 
included: 

A.  Total  RDT&E  (or  “development”)  costs:  This  metric  represents  the 
estimated  cost  of  developing  a  system  from  the  beginning  of  development 
to  the  point  at  which  it  is  ready  for  low-rate  production. 

B.  Program  Acquisition  Unit  Costs  (PAUC):  This  measure  represents  the 
expected  acquisition  cost  for  each  unit  procured,  as  determined  by 
dividing  the  sum  of  a  program’s  estimated  total  program  development, 
procurement,  and  military  construction  costs  by  the  number  of  units  to  be 
procured. 


2We  started  with  95  programs,  then  excluded  23  programs  for  which  we  were  lacking  data 
on  development  costs,  unit  costs,  or  initial  operational  capability  (IOC)  date.  We  then 
excluded  nine  programs  that  had  started  development  less  than  3  years  ago. 
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C.  Initial  Operational  Capability  (IOC)  or  equivalent  date:  IOC  is  generally 
achieved  when  some  units  or  organizations  that  are  scheduled  to  receive  a 
system  have  received  it  and  have  the  ability  to  employ  and  maintain  it. 
Where  programs  did  not  report  expected  IOC  dates,  we  substituted 
equivalent  dates  such  as  “First  Unit  Equipped”  or  “Required  Assets 
Available”  dates  reported. 

We  compared  each  program’s  initial  estimate  for  each  of  these  three 
indicators  with  the  actual/estimated  values  reported  in  its  December  2007 
SAR.  We  assigned  a  point  score  for  change  on  each  indicator  based  on  the 
following  thresholds:3 

A.  Development  (RDT&E)  cost  growth: 

•  Programs  that  reported  less  than  10  percent  development  cost 
growth  received  10  points. 

•  Programs  that  reported  at  least  10  percent,  but  less  than  35  percent 
estimated  development  cost  growth  received  5  points. 

•  Programs  that  reported  at  least  35  percent  development  cost  growth 
received  0  points. 

B.  Expected  unit  cost  (PAUC)  growth: 

•  Programs  that  reported  less  than  10  percent  growth  in  expected  unit 
costs  received  10  points. 

•  Programs  that  reported  at  least  10  percent,  but  less  than  30  percent 
growth  in  expected  unit  costs  received  5  points. 

•  Programs  that  reported  at  least  30  percent  growth  in  expected  unit 
costs  received  0  points. 

C.  Expected  initial  capability  schedule  slip: 

•  Programs  that  reported  less  than  6  months  slip  in  expected  IOC 
date  received  10  points. 

•  Programs  that  reported  at  least  6  months,  but  less  than  12  months 
slip  in  expected  IOC  date  received  5  points. 


3These  thresholds  were  determined  through  a  consideration  of  criteria  such  as  thresholds 
for  required  reporting  of  cost  and  schedule  growth  under  law  and  regulation,  historical 
average  cost  growth  for  weapons  systems  programs,  and  judgment  based  on  GAO 
experience.  These  thresholds  were  not  intended  to  directly  represent  any  thresholds  for 
allowable  cost  growth  under  law  or  DOD  regulation. 
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•  Programs  that  reported  at  least  12  months  slip  in  expected  IOC  date 
received  0  points. 

We  summed  the  scores  for  each  program  across  the  three  indicators,  to 
arrive  at  a  total  point  score  representing  our  assessment  of  the  program’s 
overall  stability.  We  categorized  each  program  as  “stable,”  “moderately 
unstable,”  or  “highly  unstable”  as  described  in  table  4. 


Table  4:  Point  Scores  for  Stability  Assessment 

Total  point  score 

GAO  assessment  of  stability 

25  or  30  points 

Stable 

10,  15,  or  20  points 

Moderately  unstable 

0  or  5  points 

Highly  unstable 

Source:  GAO. 


We  then  analyzed  the  distribution  of  stable,  moderately  unstable,  and 
highly  unstable  programs,  examining  number  of  programs,  average 
program  cost,  and  total  acquisition  dollar  value  (current  estimates) 
represented  by  programs  in  each  category.  We  also  summarized  the 
distribution  of  programs  by  age  (measured  as  years  into  development) 
among  stable,  moderately  unstable,  and  highly  unstable  programs.  Finally, 
based  on  programs’  reported  Milestone  B  and  C  dates,  we  reported 
development  cycle  time  data  for  a  subset  of  our  programs.  We  limited  this 
inquiry  to  those  programs  that  had  entered  production,  so  that  data  on 
actual  development  cycle  time  was  available.  We  excluded  ship  and 
satellite  systems  from  this  analysis,  as  system  development  start  and  end 
points  are  defined  differently  for  these  systems. 

To  more  closely  examine  factors  that  enhance  program  stability,  we  chose 
a  selection  of  five  example  programs  for  in-depth  study.  We  identified  case 
study  programs  based  on  data  from  a  variety  of  sources,  including  our 
analysis  of  programs  in  the  2008  MDAP  portfolio,  our  review  of  the 
literature  on  weapons  system  acquisitions,  including  work  conducted  by 
RAND  and  the  Defense  Acquisition  University,  and  prior  GAO  work  on 
programs  that  have  demonstrated  best  practice  approaches.  We  also 
interviewed  defense  acquisition  experts  to  learn  which  programs  are  seen 
as  role  models  among  acquisition  programs.  We  selected  the  five  case 
study  examples  using  a  criteria-based,  nongeneralizable  sample  in  order  to 
achieve  representation  across  the  military  services  as  well  as  a  variety  of 
weapon  platforms.  The  following  five  programs  were  selected  as  case 
studies:  the  Army’s  High  Mobility  Artillery  Rocket  System;  the  Air  Force’s 
Joint  Direct  Attack  Munition  and  Small  Diameter  Bomb;  and  the  Navy’s 
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Poseidon  Multi-Maritime  Aircraft  (P-8A)  and  STANDARD  Missile-6.  For 
each  case  study,  we  reviewed  key  documents,  information  from  program 
offices  on  program  managers,  and  interviewed  past  and  present  program 
officials  to  identify  key  factors  contributing  to  the  program’s  stability.  We 
also  met  with  former  senior  DOD  acquisition  officials  to  further 
understand  factors  that  stabilize  programs.  In  addition,  we  met  with 
program  officials  and  reviewed  prior  GAO  work  on  the  Air  Force’s  F-22 
Raptor  and  Global  Hawk  programs  to  better  understand  the  factors  which 
influenced  these  unstable  programs.  We  also  reviewed  prior  GAO  work 
where  we  had  identified  enablers  of  stability  in  other  programs,  including 
the  Navy’s  F/A-18E/F  Super  Hornet  and  the  Air  Force’s  F-16  Fighting 
Falcon.  To  assess  information  about  programs’  cost  estimates,  we 
compared  original  development  funding  estimates  from  programs’ 
baseline  SARs  to  development  funding  estimates  from  the  most  recent 
December  2007  SARs.  In  addition,  to  illustrate  funding  stability  for  the  P- 
8A  program,  we  compared  requested  and  received  budget  amounts  from 
budget  justification  documents. 

To  determine  how  lessons  learned  from  stable  programs  can  be  of  use  as 
DOD  implements  acquisition  reform,  we  reviewed  recent  legislative  and 
policy  changes  relating  to  defense  acquisitions  and  compared  these 
initiatives  to  our  findings  regarding  the  factors  that  enable  program 
stability. 

We  conducted  this  performance  audit  from  November  2008  to  May  2010,  in 
accordance  with  generally  accepted  government  auditing  standards.  Those 
standards  require  that  we  plan  and  perform  the  audit  to  obtain  sufficient, 
appropriate  evidence  to  provide  a  reasonable  basis  for  our  findings  and 
conclusions  based  on  our  audit  objectives.  We  believe  that  the  evidence 
obtained  provides  a  reasonable  basis  for  our  findings  and  conclusions 
based  on  our  audit  objectives. 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 

3000  DEFENSE  PENTAGON 
WASHINGTON,  DC  20301-3000 


APR  3  0  2::o 

Mr.  Michael  J.  Sullivan 

Director,  Acquisition  and  Sourcing  Management 
U.S.  Government  Accountability  Office 
441  G  Street,  NW 
Washington,  DC  20548 

Dear  Mr.  Sullivan: 

This  is  the  Department  of  Defense  (DoD)  response  to  the  GAO  Draft  Report, 
GAO-10-522,  “DEFENSE  ACQUISITIONS:  Strong  Leadership  is  Key  to  Planning  and 
Executing  Stable  Weapon  Programs.”  dated  March  23,  2010  (GAO  Code  120794). 

The  Department  is  encouraged  that  the  draft  report  cites  the  progress  we  have 
made  over  the  past  several  years  in  our  efforts  to  improve  acquisition  processes  and 
reduce  cost  and  schedule  growth.  We  are  proud  that  many  of  our  acquisition  programs 
are  performing  well,  and  we  appreciate  that  the  GAO  has  formally  recognized  several  of 
them.  We  have  instituted  several  major  changes  that  are  beginning  to  show  results.  As 
noted  in  the  draft  report,  recent  acquisition  policy  revisions  and  the  Weapon  Systems 
Acquisition  Reform  Act  (WSARA)  of  2009  are  aimed  at  starting  programs  out  right  by 
using  early  planning  and  systems  engineering,  joint  analysis  teams,  competitive 
prototyping,  configuration  steering  boards,  credible  cost  estimates,  and  program  manager 
service  agreements.  In  the  ensuing  years,  we  anticipate  improvements  in  program 
performance  measures  due  to  our  recent  acquisition  reform  initiatives.  While  the  draft 
report  does  recognize  the  gains  due  to  the  Department’s  initiatives,  the  full  breadth  of 
those  benefits  were  not  emphasized  in  the  text. 

The  Department  agrees  with  the  reasons  for  program  success,  such  as  strong 
leadership,  disciplined  program  managers,  executable  business  cases,  and  achievable 
increments  based  on  well-defined  requirements.  While  many  of  the  insights  in  the  draft 
report  are  informative,  they  are  based  upon  small,  less  costly  and  less  complicated 
programs  and  may  not  readily  scale  to  large,  software  intensive  systems  such  as  satellites, 
the  Joint  Strike  Fighter,  or  other  highly  complex  weapon  systems.  Moreover,  many  of 
the  programs  are  being  penalized  for  carrying  cost  growth  and  schedule  slips  that 
occurred  many  years  ago,  their  original  program  cost  and  schedule  estimates  are  very  old 
and  were  overly  optimistic.  The  Department  can  now  report  that  these  programs  have 
turned  the  comer  and  are  now  stable  and  performing  well. 


ACQUISITION, 
TECHNOLOGY 
AND  LOGISTICS 
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The  Department  agrees  that  the  use  of  development  cost,  unit  cost  and  delay  in 
IOC  are  useful  measures  of  program  performance;  however,  they  alone  are  not  the  only 
measures  of  stability.  The  draft  report  defines  stable  and  unstable  programs  not  in  terms 
of  historical  outcomes,  but  rather  as  whether  programs  "are  relatively  on-track  to  meet 
cost  and  schedule  targets  as  defined  at  program  development  start.”  This  definition  is 
valid  when  operational  requirements  remain  static  throughout  the  program  lifecycle.  In 
some  cases,  the  Warfighter’s  needs  change  through  time,  and  DoD  has  an  obligation  to 
provide  the  best  weapon  systems  to  achieve  U.S.  military  objectives.  The  Department 
owes  it  to  the  nation  to  be  able  to  adapt  to  a  changing  international  threat  environment. 
This  strategic  agility  and  its  associated  enhanced  system  functionality  do  impact  cost  and 
schedule.  Managing  programs  on  the  cutting  edge  of  science,  technology,  and 
engineering  does  carry'  risk  with  it.  For  these  reasons,  it  is  not  always  useful  to  assess 
acquisition  programs  relative  to  the  initial  conditions  at  program  inception. 

I  thank  you  and  your  staff  for  working  with  the  Department  to  improve  the 
information  flow  between  our  organizations  and  to  develop  more  meaningful  metrics  in 
this  area.  I  look  forward  to  continuing  to  improve  the  acquisition  process  to  more 
effectively  and  efficiently  deliver  products  to  our  customers,  and  we  need  to  continue  to 
develop  better  metrics.  The  Department  looks  forward  to  working  with  the  GAO  in  both 
of  these  important  endeavors. 

The  Department  appreciates  the  opportunity  to  comment  on  the  draft  report. 
Additional  comments  are  provided  as  an  enclosure  to  this  letter.  My  point  of  contact  for 
this  effort  is  Mr.  Joseph  Alfano,  703-697-3343. 


Sincerely, 


cjff ' 


Dr.  Nancy  L.  Spruill 
Director, 

Acquisition  Resources  &  Analysis 


Enclosure: 
As  stated 
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ENCLOSURE 

GAO  DRAFT  REPORT  DATED  MARCH  23,  2010 
GAO-10-522  (GAO  CODE  120794) 

“DEFENSE  ACQUISITIONS: 

STRONG  LEADERSHIP  IS  KEY  TO  PLANNING  AND  EXECUTING  STABLE 
WEAPON  PROGRAMS” 

DEPARTMENT  OF  DEFENSE  COMMENTS 
ON  THE  DRAFT  REPORT 

Applications  of  Lessons  Learned 

•  The  draft  report  does  not  emphasize  recent  initiatives  that  the  Department 
undertook  to  enhance  program  acquisition  management.  For  instance,  page  4  of 
the  draft  report  states  “...programs  enter  the  acquisition  process  without  a  full 
understanding  of  requirements...”  which  reflects  the  “sins  of  the  past.”  The 
programs  that  failed  to  properly  understand  requirements  were  among  the  oldest  in 
the  portfolio.  The  Department  has  since  placed  a  much  greater  emphasis  on 
managing  requirements.  Additionally,  given  the  uncertain  and  evolving  military 
threats  that  confront  our  nation,  such  a  high  standard  for  requirements  may  be 
unachievable.  Because  the  nature  of  the  threats  is  not  static  throughout  the  life  of 
a  weapon  system,  neither  are  the  program  requirements.  The  Department  agrees 
that  resisting  new  requirements  is  important,  and  we  have  implemented 
configuration  steering  boards  to  manage  requirements  growth;  however,  the 
Department  must  also  be  responsive  to  the  Warfighters’  future  needs. 

•  The  draft  report  implies  that  PM  tenure  is  a  significant  factor  in  program  stability. 
The  Department  agrees  that  while  it  is  a  contributing  factor,  characterizing  PM 
tenure  as  an  average  is  misleading.  Specifically,  on  page  16,  the  draft  report  states 
that  the  average  program  manager  (PM)  tenure  is  1.4  years.  By  design,  PMs  serve 
in  their  positions  for  three  years,  so  at  any  point  in  time  the  population  of  PMs 
should  be  half-way  through  their  three  year  tenure:  statistically  1.5  years.  The 
draft  report  implies  that  this  “snapshot  in  time”  average  PM  tenure  of  1 .4  years 
represents  the  total  time  PMs  serve  in  their  positions.  Since  the  full  tenure  period 
for  a  PM  is  three  years,  this  is  clearly  not  the  case.  Since  this  is  not  the  first  time 
we  have  disagreed  on  this  statistic,  I  would  offer  to  collaborate  with  you  to  sample 
the  PMs  and  have  a  shared  database  from  w'hich  to  work. 

Assessment  of  Programs  vis-a-vis  cost  and  schedule  growth 

•  The  evaluation  criteria  are  subjective.  Page  3  of  the  draft  report  contains  the 
evaluation  criteria  and  concedes  that  the  criteria  are  not  traceable  to  legislation  or 
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regulations  used  to  report  cost  and  schedule  growth.  The  draft  report's  evaluation 
criteria  are  ad  hoc  and  internally  inconsistent:  the  upper  cut-off  for  RDT&E  cost 
growth  is  35%,  while  the  equivalent  threshold  for  program  acquisition  unit  cost 
(PAUC)  growth  is  30%.  When  the  study’s  criteria  are  applied  to  the  five 
programs  identified  as  exemplars,  two  of  them  (HIMARS  and  JDAM )  do  not  rank 
as  stable  programs. 

•  The  study  used  nonstandard  cost  and  schedule  data.  The  Objectives,  Scope  and 
Methodology  section  states  that  “other  program  data”  supported  the  study.  The 
Department  has  been  unable  to  reproduce  the  study  results  using  Selected 
Acquisition  Report  (SAR)  data. 

•  The  Department  recommends  using  evaluation  criteria  based  on  accepted 
standards,  legislation,  and  regulations.  The  table  below  relates  the  DoD  growth 
parameters  to  enacted  legislation.  For  IOC  slip,  the  cutoffs  are  12  months  and  24 
months  total  growth  beginning  with  the  IOC  defined  at  Milestone  B. 


Growth 

Parameter 

DoD  Criteria 
(moderate  /  high  growth) 

U.S.  Code  (Title  10) 

RDT&E 

+30%  /  +50% 

N/A 

PAUC 

+30%  /  +50% 

+30%  /  +50%  (Nunn-McCurdy) 

IOC 

_ 

+12  months  /  +24  months 
since  milestone  B 

+6  months  since  prior  quarterly  SAR 
(10  USC  2432) 

•  Using  the  Dec  2007  SAR  data  and  the  above  evaluation  criteria,  the  portfolio 
growth  profile  appears  in  the  following  table. 


Category 

Programs  by 
Category 

Total  Acquisition 
Cost  by  Category 

Low  Growth 

40% 

15% 

Moderate  Growth 

47% 

72% 

High  Growth 

13% 

13% 
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Michael  J.  Sullivan,  (202)  512-4841  or  sullivanm@gao.gov 
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GAO’s  Mission 

The  Government  Accountability  Office,  the  audit,  evaluation,  and 
investigative  arm  of  Congress,  exists  to  support  Congress  in  meeting  its 
constitutional  responsibilities  and  to  help  improve  the  performance  and 
accountability  of  the  federal  government  for  the  American  people.  GAO 
examines  the  use  of  public  funds;  evaluates  federal  programs  and  policies; 
and  provides  analyses,  recommendations,  and  other  assistance  to  help 
Congress  make  informed  oversight,  policy,  and  funding  decisions.  GAO’s 
commitment  to  good  government  is  reflected  in  its  core  values  of 
accountability,  integrity,  and  reliability. 

Obtaining  Copies  of 
GAO  Reports  and 
Testimony 

The  fastest  and  easiest  way  to  obtain  copies  of  GAO  documents  at  no  cost 
is  through  GAO’s  Web  site  (www.gao.gov).  Each  weekday  afternoon,  GAO 
posts  on  its  Web  site  newly  released  reports,  testimony,  and 
correspondence.  To  have  GAO  e-mail  you  a  list  of  newly  posted  products, 
go  to  www.gao.gov  and  select  “E-mail  Updates.” 

Order  by  Phone 

The  price  of  each  GAO  publication  reflects  GAO’s  actual  cost  of 
production  and  distribution  and  depends  on  the  number  of  pages  in  the 
publication  and  whether  the  publication  is  printed  in  color  or  black  and 
white.  Pricing  and  ordering  information  is  posted  on  GAO’s  Web  site, 
http://www.gao.gov/ordering.htm. 

Place  orders  by  calling  (202)  512-6000,  toll  free  (866)  801-7077,  or 

TDD  (202)  512-2537. 

Orders  may  be  paid  for  using  American  Express,  Discover  Card, 

MasterCard,  Visa,  check,  or  money  order.  Call  for  additional  information. 

To  Report  Fraud, 
Waste,  and  Abuse  in 
Federal  Programs 

Contact: 

Web  site:  www.gao.gov/fraudnet/fraudnet.htm 

E-mail:  fraudnet@gao.gov 

Automated  answering  system:  (800)  424-5454  or  (202)  512-7470 

Congressional 

Relations 

Ralph  Dawn,  Managing  Director,  dawnr@gao.gov,  (202)  512-4400 

U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7125 
Washington,  DC  20548 

Public  Affairs 


Chuck  Young,  Managing  Director,  youngcl@gao.gov,  (202)  512-4800 
U.S.  Government  Accountability  Office,  441  G  Street  NW,  Room  7149 
Washington,  DC  20548 
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